Repository: statsmodels/statsmodels

Issue:
BUG tukeyhsd nan in group labels
while checking and fixing #1561 
using example notebook from comment in that issue

The labels with integer group labels show nans when using Pandas DataFrame/Series for data.
The first case copied below uses recarrays and is ok, the second (In [7]) uses pandas DataFrame and shows nans.

The example with string/bytes labels looks completely messed up with pandas DataFrame with more lines than pairs and nans as group/level labels (not copied here)

`pandas.__version__`  is '0.12.0'

```
In [6]:

mc = multi.MultiComparison(dta['Rust'], dta['Brand'])
res = mc.tukeyhsd()
print(res.summary())
Multiple Comparison of Means - Tukey HSD,FWER=0.05
===============================================
group1 group2 meandiff  lower    upper   reject
-----------------------------------------------
  1      2      46.3   43.3155  49.2845   True 
  1      3     24.81   21.8255  27.7945   True 
  1      4     -2.67   -5.6545   0.3145  False 
  2      3     -21.49  -24.4745 -18.5055  True 
  2      4     -48.97  -51.9545 -45.9855  True 
  3      4     -27.48  -30.4645 -24.4955  True 
-----------------------------------------------
```

```
In [7]:

mc = multi.MultiComparison(dta1['Rust'], dta1['Brand'])#.values)
res = mc.tukeyhsd()
print(res.summary())
Multiple Comparison of Means - Tukey HSD,FWER=0.05
===============================================
group1 group2 meandiff  lower    upper   reject
-----------------------------------------------
 1.0    nan     46.3   43.3155  49.2845   True 
 1.0    nan    24.81   21.8255  27.7945   True 
 1.0    nan    -2.67   -5.6545   0.3145  False 
 nan    nan    -21.49  -24.4745 -18.5055  True 
 nan    nan    -48.97  -51.9545 -45.9855  True 
 nan    nan    -27.48  -30.4645 -24.4955  True 
-----------------------------------------------
```



Discussion:
bug: I use n`p.asarray` on `groups`, but then use the original in the call to np.unique

```
        self.groups = np.asarray(groups)

        # Allow for user-provided sorting of groups
        if group_order is None:
            self.groupsunique, self.groupintlab = np.unique(groups,
                                                            return_inverse=True)
```

`groupsunique` is a pandas.Series not an ndarray

no idea which pandas version I was using that worked in the gist notebook
http://nbviewer.ipython.org/gist/josef-pkt/10000517

unique on pandas.Series for groups produces nonsense

```
n [27]:

res2.groupsunique
Out[27]:
Treatment
29            medical
20           physical
8              mental
0                 NaN
3              mental
19           physical
10             mental
15           physical
Name: Treatment, dtype: object
```

using 
`self.groups = groups = np.asarray(groups)`  
in `MultiComparison.__init__` to avoid pandas internally fixes the bugs that show up in the notebook.


Resolving patch:
diff --git a/statsmodels/sandbox/stats/multicomp.py b/statsmodels/sandbox/stats/multicomp.py
--- a/statsmodels/sandbox/stats/multicomp.py
+++ b/statsmodels/sandbox/stats/multicomp.py
@@ -784,7 +784,7 @@ class MultiComparison(object):
 
     def __init__(self, data, groups, group_order=None):
         self.data = np.asarray(data)
-        self.groups = np.asarray(groups)
+        self.groups = groups = np.asarray(groups)
 
         # Allow for user-provided sorting of groups
         if group_order is None:


Test patch:
diff --git a/statsmodels/stats/tests/test_pairwise.py b/statsmodels/stats/tests/test_pairwise.py
--- a/statsmodels/stats/tests/test_pairwise.py
+++ b/statsmodels/stats/tests/test_pairwise.py
@@ -227,6 +227,18 @@ def test_table_names_custom_group_order(self):
             assert_((first_group, second_group) == expected_order[i - 1])
 
 
+class TestTuckeyHSD2Pandas(TestTuckeyHSD2):
+
+    @classmethod
+    def setup_class(self):
+        super(TestTuckeyHSD2Pandas, self).setup_class()
+
+        import pandas
+        self.endog = pandas.Series(self.endog)
+        # we are working with bytes on python 3, not with strings in this case
+        self.groups = pandas.Series(self.groups, dtype=object)
+
+
 class TestTuckeyHSD2s(CheckTuckeyHSDMixin):
     @classmethod
     def setup_class(self):