BUG: series timedelta arithmetic is not being converted to ns with numpy 1.6
State: closed

closes #4135.


Labels:

Comments:
Does the Travis fail mean that it will not be considered for merge?

no that's unrelated

just didn't get to it

this was failing for numpy running on python 2.6 having to do with `timedelta64`'s arguments need to figure out what is up with that.

bug here is a bit more subtle because of numpy 1.6.x's different timedelta/datetime API.

is Travis having numpy 1.6 with Python 2.6 only? Because I was using numpy 1.6 with Python 2.7 for, I believe, years! So that would be a much more realistic use case.

@michaelaye see https://github.com/pydata/pandas/pull/4153#issuecomment-20584817

short is answer is no, it's not :)

ok, so that means the fail with the 2.6 env (or better to say locale?) can not have anything to do with numpy 1.6.1, because it works in 2.7, right?

yes that's a different issue, as i pointed out above. before i was passing 2 args to timedelta64 which is invalid in numpy 1.6.x, now there's something else going on.

@michaelaye no, in fact, i'm almost 100% sure it has something to do with 1.6.x. look at how the tests are run....
- 2.6 runs all tests that don't require full dependencies and uses 1.6.1: FAIL
- 2.7 without LOCALE runs all tests with full deps and uses 1.7.1: PASS
- 2.7 with locale runs slow and not network tests, and uses 1.6.1: PASS (because it doesn't run my new test)
- 3.2 same as 2.6 but with full deps and numpy 1.6.2: FAIL
- 3.3 numpy 1.7.1: PASS

1.6.2 IS different (buggy basically).....will have to look at this

this will help:

http://docs.scipy.org/doc/numpy-dev/reference/arrays.datetime.html#differences-between-1-6-and-1-7-datetimes

oh, but the 3.2 env did work before, no? I thought I saw before that 2.6 env was the only one failing, so that must be your new test, I guess. (I am a noob about Travis, need to learn this ...)

yep. before it was failing because you can't pass units to old timedelta (raising the standard `TypeError` when you call a function with more arguments than it has formal parameters), so i just manually did it by taking the dot product of the unit number of microseconds with the coefficients in my test (old timedelta doesn't support lower than microseconds), now pandas is raising

just to make it even harder
python 3 is completely different 
treats timedelta more like ints

joy

i think the type testing function is being too strict...checking it out now

btw, we really should try and reduce the build time of the cython stuff by using fused types

That's actually a question I had on my mind for months: I see that homebrew can build using all cores, but the pandas built ever only uses 1 core. How can one improve that?

i think i fixed the arith issue, i wasn't converting to timedelta in the test...we'll see what happens

@michaelaye not sure about that...i don't think you can parallelize that (easily) since `setup.py` is what is controlling the compilation, so the gcc process has python as it's parent process and thus i think you'd need some way to tell python that it's ok to start up multiple gcc processes on independent files

but really i was referring to algos and things made by `generated.py` because that's where most of the bottleneck is and most of the repetition across types. right now pandas is generating types manually where fused types are similar to templates they allow you to write the code once and let the compiler (Cython) take care of generating the code  for the different types. what's kind of cool is that you can use the function like a dict where the keys are types and the values are the specific function for that type which might allow you to fine tune if you needed to, at a higher level

arg now it's failing because of something that looks minor. can't look at it right now ...

update:

going to get to the bottom of why this works this weekend and if everything makes sense i'll write it up here and merge

closing in favor of #4534

thanks