Weighted mean
State: closed


Labels:
name: Enhancement
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
@wesm ok to introduce fused types yet? Guess it's not strictly necessary, but would like to enable it so dt64 ints can be used as weights

is this v0.11 material?

great feature! when do you think you will merge this?

@changhiskhan I can include float32 in the fuse dtypes for compat would be great

@jreback that would be great thanks.

@gerigk it's just pending review (and jreback's additions). It'll definitely make it in 0.12 if not before.

@changhiskhan out of curiosity is there any difference (perf/ease of use) for fused types vs type generation (eg generated.pyx)

Haven't done a performance comparison but I'd be surprised if there was a substantial difference there. 
fused types definitely easier to use and more natural, at least more so than the current code gen stuff in pandas.

On Mar 4, 2013, at 8:41 AM, jreback notifications@github.com wrote:

> @changhiskhan out of curiosity is there any difference (perf/ease of use) for fused types vs type generation (eg generated.pyx)
> 
> —
> Reply to this email directly or view it on GitHub.

I don't think there is. I would suggest migrating as much of the generated.pyx code to use fused types as soon as feasible. There are some NA-handling issues where it may not be possible to migrate everything. 

Aren't compilation times longer?

We're currently doing "fused types by hand". I think the compilation times may actually be shorter because the number of Python entry points will be a lot smaller

@changhiskhan 
I rebased a copy of your branch onto current master....2 failures
can you take a look?

```
[sheep-jreback-~/pandas] nosetests pandas/tests/test_frame.py 
.........................................................................................................................................................................................SSS...........................................................................................................................................................................................................................S....................S...................F......
======================================================================
FAIL: test_weighted_mean_1d_weights (pandas.tests.test_frame.TestDataFrame)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/home/jreback/pandas/pandas/tests/test_frame.py", line 7515, in test_weighted_mean_1d_weights
    inject_skipna=True)
  File "/mnt/home/jreback/pandas/pandas/tests/test_frame.py", line 7680, in _check_stat_op
    check_dtype=False)  # HACK: win32
  File "/mnt/home/jreback/pandas/pandas/util/testing.py", line 164, in assert_series_equal
    assert_almost_equal(left.values, right.values, check_less_precise)
  File "/mnt/home/jreback/pandas/pandas/util/testing.py", line 106, in assert_almost_equal
    assert_almost_equal(a[i], b[i], check_less_precise)
  File "/mnt/home/jreback/pandas/pandas/util/testing.py", line 137, in assert_almost_equal
    1, a / b, decimal=decimal, err_msg=err_msg(a, b), verbose=False)
  File "/usr/local/lib/python2.7/site-packages/numpy/testing/utils.py", line 460, in assert_almost_equal
    raise AssertionError(msg)
AssertionError: 
Arrays are not almost equal to 5 decimals expected -0.47729 but got nan

----------------------------------------------------------------------
Ran 455 tests in 4.492s

FAILED (SKIP=5, failures=1)
[sheep-jreback-~/pandas] nosetests pandas/tests/test_series.py
...........................................................................................S..........................................................................................................................................S.............E............
======================================================================
ERROR: test_weighted_mean (pandas.tests.test_series.TestSeries)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/home/jreback/pandas/pandas/tests/test_series.py", line 1363, in test_weighted_mean
    self._check_stat_op('mean', wgt_mean, weights=weights)
  File "/mnt/home/jreback/pandas/pandas/tests/test_series.py", line 1512, in _check_stat_op
    testit()
  File "/mnt/home/jreback/pandas/pandas/tests/test_series.py", line 1503, in testit
    assert_almost_equal(float(f(s)), float(alternate(s.values)))
  File "/mnt/home/jreback/pandas/pandas/tests/test_series.py", line 1358, in wgt_mean
    mask = notnull(weights).mul(notnull(data)).dropna().astype(bool)
  File "/mnt/home/jreback/pandas/pandas/core/series.py", line 322, in f
    raise ValueError('Lengths must be equal')
ValueError: Lengths must be equal

----------------------------------------------------------------------
Ran 257 tests in 1.208s

FAILED (SKIP=2, errors=1)
[sheep-jreback-~/pandas] 
```

what is the status here? is it safe to use?

will this feature really make it into 0.13? I seem to be the only one really interested for quite some time now ;-)

i'd definitely be interested in this as well.

pushing to 0.14, needs some TLC

still planned for 0.14? I am not technical enough to help here but I would be a huge fan of not having to write ugly weighted mean workarounds anymore :)

would need someone to champion this.....

I'll take it. I'm not a C coder, so if anyone else wants it they can have it.

closing as stale.

@TomAugspurger if you want to pick this up, by all means