Repository: statsmodels/statsmodels

Issue:
BUG: `sandwich_covariance.group_sums` raises avoidable MemoryError
``` python
import pandas as pd
import numpy as np
from numpy.random import randn, randint

import statsmodels.api as sm

N = 1000000
Ng = 100000
df = pd.DataFrame({'y': randn(N), 'a': randn(N), 'b': randn(N),
                   'group': randint(Ng, size=N)})
# Make sure the group ID is an int
df['group'] = df['group'].astype(np.int32)
# And has a big range to screw up numpy.bincount
df.loc[0, 'group'] = 100000000

reg = sm.OLS.from_formula('y ~ a + b', df).fit()
# both of these variants run into the same problem
if 1 == 1:
    clust = reg.get_robustcov_results('cluster', groups=df['group'])
else:
    clust = reg.get_robustcov_results('cluster', groups=df['group'].values)
```

The use of `np.bincount` in `stats.sandwich_covariance.group_sums` raises a MemoryError is the `group` array has large values and is an `int` (since `bincount` spans the whole range of values, it's not the size of the `groups` array, but the actual range). Floats and anything without a `dtype` attribute gets re-indexed, but these don't. Again, should be a real quick fix (and easy to work around, as I did all day today), but wanted to check if there some reason I wasn't aware of for why the int's aren't getting re-indexed.



Discussion:
Thanks for the report and feedback. There is insufficient testing for interface problems in this code, since I spend most of my time on unit tests to match small sample options with Stata.

The integer version is supposed to be the "fast track" definition. If we have to repeatedly calculate this on the same or similar problems, then it is wasteful to do the same factor to integer conversion for the groups each time. In this case, it's better to predefine or convert groups in advance to correspond to `np.arange(n_groups, int)`.

The question is how to distinguish fast track integers, from integer groups that mess up `np.bincount` as in this example.
It would be helpful if you have improvement suggestions that don't completely remove any fast track possibilities.

I haven't looked at the details in some time, but the sandwich code needs another round of review and refactorings.

I hadn't considered the computational cost of always checking to be that large, since this function would be called only a handful of times per regression and it's fairly quick (memory usage may be a different story). Though I suppose bootstrapping with lots of reps could suffer from this.

My only other thought would be to try-except MemoryError and re-index the groups only when the Memory Error is raised. If that sounds okay, I'll put it together. It is probably fairly rare bug, maybe a more informative error message there would be sufficient if you think other try-except strategy might be too slow.

Even small duplicate or redundant calculations will eventually add up, and I don't think this is so cheap.

After thinking about this: I think the easiest cheap way to prevent this kind of problems coming from bincount, is to reindex if the groups.max() is too large, say 0.5 \* nobs or 0.75 \* nobs, or equal to nobs.
The first two would be reasonable limits on the number of cluster, the last is the upper bound and I think it would be allowed to have single observation clusters. (Should correspond to HC1 ? or proportional).
In some cases where we run the model on only a subset of observations, we might have a predefined groups with larger group numbers than nobs.

I think we reindex with `nobs` as threshold.  code should be in grouputils

AFAICS, the version of group handling that is used by the current cov_cluster function, does not use `grouputils`, it just has a call directly to np.unique in `stats.sandwich_covariance.cov_cluster`.

`grouputils`, `Group` is currently only used to get the intersection group in the 2-cluster case.

> ... the easiest cheap way... is to reindex if the groups.max() is too large, say 0.5 \* nobs or 0.75 \* nobs

This would work, though it would actually catch more cases than necessary. The problem with `bincount` is purely nominal since it makes a 1D array as long as the largest element of its argument:

``` python
In [1]: np.bincount([100]).shape
Out[1]: (101L, )
In [2]: np.bincount([10000000000]).shape
MemoryError                               Traceback (most recent call last)
<ipython-input-13-2955d84d9b2e> in <module>()
----> 1 np.bincount([10000000000]).shape

MemoryError:
```

The within-group sum returned from using the weights argument is obviously smaller, but I think it is still doing a similar expansion behind the scenes.

A hard cap on `groups.max()` I think would be more efficient than comparing it to `nobs`. Maybe 50,000,000, half of what is above? I don't know. (I can actually get my original example to run if I close everything else on my computer. The OS tries to kill the python process, but the script finishes.)

It's slightly more arbitrary, but should allow more cases to pass through without re-indexing.

The problem with the hard cap is that it depends on the computer and it's load, and I would be surprised as user if my program suddenly takes a huge amount of memory.
(I have only about 1 GB RAM left, the rest is already in use, and most likely most of the 1GB are fragmented and cannot be used for a single array that needs a contiguous block of memory.)

We can increase the threshold to a multiple of nobs, like `2*nobs`, which would let many `groups` unchanged that have integers with holes.

Sounds good, `2*nobs` it is.

good, 
Do you want to prepare the PR?  I'd like to get this fixed for 0.7.

Yes, I'll try to bang this out tonight.

On Tue, Jan 27, 2015 at 11:17 AM, Josef Perktold notifications@github.com
wrote:

> good,
> Do you want to prepare the PR? I'd like to get this fixed for 0.7.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/statsmodels/statsmodels/issues/2169#issuecomment-71675663
> .

Thanks


Resolving patch:
diff --git a/statsmodels/stats/sandwich_covariance.py b/statsmodels/stats/sandwich_covariance.py
--- a/statsmodels/stats/sandwich_covariance.py
+++ b/statsmodels/stats/sandwich_covariance.py
@@ -102,6 +102,7 @@
 
 """
 from statsmodels.compat.python import range
+import pandas as pd
 import numpy as np
 
 from statsmodels.tools.grouputils import Group
@@ -433,6 +434,11 @@ def group_sums(x, group):
     '''
 
     #TODO: transpose return in group_sum, need test coverage first
+
+    # re-label groups or bincount takes too much memory
+    if np.max(group) > 2 * x.shape[0]:
+        group = pd.factorize(group)[0]
+
     return np.array([np.bincount(group, weights=x[:, col])
                             for col in range(x.shape[1])])
 
diff --git a/statsmodels/tools/grouputils.py b/statsmodels/tools/grouputils.py
--- a/statsmodels/tools/grouputils.py
+++ b/statsmodels/tools/grouputils.py
@@ -99,6 +99,11 @@ def group_sums(x, group, use_bincount=True):
         raise ValueError('not implemented yet')
 
     if use_bincount:
+
+        # re-label groups or bincount takes too much memory
+        if np.max(group) > 2 * x.shape[0]:
+            group = pd.factorize(group)[0]
+
         return np.array([np.bincount(group, weights=x[:, col])
                          for col in range(x.shape[1])])
     else:


Test patch: