CLN/ENH/BLD: Remove need for 2to3 for Python 3.
State: closed

Fixes #4375 and #4372.

Many changes to make codebase compatible in 2 and 3. For `range`, `zip`, `map`, etc. tried to favor iterators over needing to use lists.

Changes:
- No more 2to3 in setup.py
- merges util/compat and util/py3compat into pandas/compat
- incorporates useful parts of the six library into compat (+ adds SIX to LICENSES)
- defaults to using iterators in both Python 2 and Python 3 for range, zip, map, and filter
- Adds lrange, lzip, lmap, and lfilter, which wrap corresponding iterator methods with lists
- Improved type checks (where appropriate --> many need to be rigid to work with pandas)
- various other utilities to be Py2/3 compatible (iteritems, iterkeys, itervalues, etc.)
- deprecates iterkv with a warning (no longer necessary because library no longer uses 2to3)
- compatibility wrapper around `dateutil` for handling unicode when version <= 2.0 + 1.5 dateutil build in Travis

Switches everything to use special iteritems, so can deprecate as discussed in #4372.

This iteritems use "iteritems" method if available and otherwise uses items.

Checklist of packages to run through 2to3 to check:
- [x] compat
- [x] core
- [x] io
- [x] rpy
- [x] sandbox
- [x] sparse
- [x] src (python files only)
- [x] stats
- [x] tests
- [x] tools
- [x] tseries
- [x] util

Other tasks:
- [x] alias `__bool__` to `__nonzero__`


Labels:

Comments:
I think GitHub might explode if I click on the files changed tab :boom:

I think `filter` is also an iterator in Python 3 IIRC

@cpcloud it is...still coming up on my list :P

@cpcloud actually, only used in one place in library and it's used as an iterator anyways. still need to check for itertools changes.

great. i think i might use `filter` in `eval-3393`, i'll check

@cpcloud just put `from six.moves import filter` at the top of the file and you'll be fine. It just uses itertools.ifilter under the hood, so slap a list on it otherwise. Very simple

cool thanks! pandas is going to rule the world soon.

i hope python 3 testing is going to be faster after this .... those 2to3 runs seem like they take FOREVER

@cpcloud I _think_ it makes the entire test suite a little faster...not sure.

nice looks like travis build times are 3-4 mins faster that is cool

@cpcloud does Cython have any changes for Python 3?

I believe all of those issues are taken care of during translation from the Cython to C. I don't see why any Cython would need to be changed. Are you getting compilation errors?

@cpcloud guess I can empirically see that that's not true...just gone through so many changes (and I can't get py3 pandas to work on my mac).

@cpcloud I'm getting all these weird stata reader errors. Do you have any ideas on what might be causing them?

@jtratner can u link me to a build with those errors?

@cpcloud check out the first few errors on here: https://travis-ci.org/jtratner/pandas/jobs/9563671

have you tried stepping with a debugger?

@cpcloud I've been having trouble getting pandas with Python 3 to install on my mac, which is why I haven't been able to step through.

@cpcloud but eventually I'll figure out how to get it to work

ah ok... it's weird because i think you actually have to `setup.py install` it for `2to3` to run which is annoying whenever you make a change...you have to keep doing it

@cpcloud well, the current build has 2to3 disabled, so you don't actually need to do that.  I'm trying a separate build with 2to3 enabled to see if it resolves the problem.

indeed, i hadn't pulled down your PR in my py3.2 venv yet :smile: 

anyway my dev version works so just keep at it i think you'll get it ... i would offer to clone my env but i'm not using mac

@cpcloud okay, pretty sure I can figure it out. It doesn't fail with any of the stata errors with 2to3, so that means I just have to apply some of those fixes and it will work :).

for some reason the `path_or_buf.read()` calls are not reading anything...

(in `_read_header`)

@cpcloud thanks. And I found all the non-stata errors are resolved by 2to3'ing pandas/core.

ahh so the problem is actually not in `_read_header` you need to convert the calls to `np.where(map(...))` to `np.where(list(map()))` then you're golden...i just tried it, it works

that's in the `data` method

i love getting to the bottom of things!