BUG: Series dtype casting to platform numeric (GH #2751)
State: closed

use int64/float64 defaults in construction when dtype is not specified
this removes a platform dependency issue that caused dataframe and series to
have different dtypes on 32- bit

fixes issues raised in PR #2837 


Labels:

Comments:
just curious, does this still work without overflow if the list explicitly contains longs? (i.e. `df=p.DataFrame({'a' : [2**31,2**31+1]})`)

that first commit blew up....was always using platform int, but that's not exactly right, trying asarray (which is what DataFrame does)...

@stephenwlin

can you take a look with either of my commits? these should work
I am in theory duplicating what DataFrame([1,2]) does for DataFrame({'a' : [1,2]})
(all tests pass on 64-bit), something is forcing int64 elsewhere and so a bunch of tests fails

sure. i guess you're not developing on 32-bit so you don't have an easy way to test this? i'll look into it after i finish with something else...

nope....64-bit...linux....no easy way to even put a dev build (i did it on windows, but then all kind of things are hard :)

I guess it comes down to:

should `DataFrame([1,2])` be `int32` on 32-bit and `int64` on 64-bit, or always `int64`
before the dtypes change this was very clear, always `int64`, now not so sure

@wesm, @changhiskhan any thoughts?

latest commit only breaks 4 tests on 32-bit!
I changed DataFrame([1,2]) to do what series does now (which is upconverts lists to int64 I think)

I need to rebase this...later..

Yeah, I agree that it's better to use the same code path throughout for consistentcy. Not entirely sure that upcasting to `int64` always is the best, but it's the path of least resistance for now.

I found some other issues in `maybe_convert_objects` (#2845) which I've fixed in a branch but I haven't changed that part of the behavior, pending a decision from someone else on what should be done.

ok..thanks....let me know if you have any success with this branch.....I cannot easily debug this....but think that unless explicty types, ints get `int64` and floats get `float64` regardless of architecture (which is how numpy defaults). This, I is also what you get when you convert to objects and then `maybe_convert_objects`. Bottom line is that `DataFrame([1,2])` should be `int64`

@jreback, you can setup a VM for 32 bit testing with `vagrant`, ubuntu has 
precise32 boxes all ready, just put the source tree in a shared directory and go.

oh...this looks nice....any particular box you think?

@y-p ignore my last - didn't read it 

http://www.dejonghenico.be/unix/setup-vagrant-and-small-quick-start

setup was pretty easy

@y-p any chance u have a chef recipe laying around?

sorry. I think once you create the box, you can edit the shared folders 
through the vbox gui, if that's what you're after.

it's more like a Travis install script to setup the base environment
no worries
thanks

I wouldn't bother with chef, just use virtualenv for 3.x, like you would on your local box.
tox used to work, but bugged out at some point during the distribute/virtualenv mess.

I just did manual installation 
all worked
(pip was a little funny - I had to install everything via apt-get)
but all ok and got tests to run

now to debug!

@stephenwlin ok I fixed this up
pls check this out and lmk if anything looks weird

API change on 32- bit is that
DataFrame([1,2]) is now int64 (and not int 32)
what was this in 0.10.1?

{'a' : [1,2] } will still be int64

Looks good but I'm curious why you're calling `_possibly_convert_objects` with three non-default arguments just to end up getting to `values = lib.list_to_object_array(values)` and `values = lib.maybe_convert_objects(values)` instead of just calling the latter two directly (as was originally done in `_sanitize_array`). No big deal, but it just looks kind of ugly with the calls spread out over four different lines.

Also, from looking at the test it seems that now `DataFrame([1, 2])` and `DataFrame({'a' : [1, 2]})` yield int64 but `DataFrame({'a' : 1})` yields platform int...maybe that should change too?

also "platform independent manor" typo in doc/source/v0.11.0.txt

did all of these 3 cases yield int64 in 0.10.1 on 32-bit?

what's wrong with 'platform independent manor'?

getting closer!

@stephenwlin FYI I moved _dtype_from_scalar to common
I think it makes sense to combine this with your _maybe_promote (and _try_cast)
maybe needs another argument - and have to fix API 

platform independent "manner", right?

0.10.0 good enough?

``` python
In [1]: import pandas as p

In [2]: p.__version__
Out[2]: '0.10.0'

In [3]: p.DataFrame({'a': 1}, index=[0]).dtypes
Out[3]: a    int64

In [4]: p.DataFrame({'a': [1, 2]}).dtypes
Out[4]: a    int64

In [5]: p.DataFrame([1, 2]).dtypes
Out[5]: 0    int64
```

yes

ok so the behavior I am suggesting will not actually change the API - that is good

duh!

thxs

On Feb 13, 2013, at 11:41 AM, stephenwlin notifications@github.com wrote:

> platform independent "manner", right?
> 
> —
> Reply to this email directly or view it on GitHub.

finally all tests pass!

@stephenwlin pls take a look when you have a chance....

It looks great! But I changed the signature of `_maybe_promote` to return a (possibly modified) `fill_value` (it makes sense when you see it) in my "stephenwlin/opt-take-2" branch, so it'll conflict during a merge.

If you want to proactively fix it, I've already resolved the differences in "stephenwlin/dtypes_bug" (currently pointing to f74571f7a613d1f971c99cac7a53ee077b1582f6), so if you "git reset" to that you'll have all your changes merged with the `_maybe_promote` signature change (but not the rest of "stephenwlin/opt-take-2"), rebased appropriately so we'll merge cleanly later. (Basically, I made a minimal commit off master with just the signature change and then rebased the two branches off that, so they share the same history)

If you want to look at the differences to make sure they're ok, just take a look at the commit 05a4991f014f7ed55b6d8270f06c3c554f05189b, which shows only the differences between the current state of your branch (at 3cb91f09c5a710c589d365e3dfea20501b09b978) and "stephenwlin/dtypes_bug", without any rebasing.

If you'd rather not bother, that's ok too...I can always do the conflict resolution again later after one of our PRs gets merged.