nltk/nltk — Issue #80: NLTK 3.0
Author: alexrudnick
Created: 2012-01-17 06:25:37+00:00
State: open

This is the generic issue concerning support for Python 3.0. Planning will
begin once NLTK 2.0 is released. We'll probably create a 3.0 fork and ask
for offers of help. To volunteer, please add a comment to this issue.
Please star this issue to hear of progress.

Migrated from http://code.google.com/p/nltk/issues/detail?id=440

earlier comments

jg07024 said, at 2009-09-04T01:21:29.000Z:

Count me in.

John Graves
PhD Student
Auckland University of Technology
vhy4743@aut.ac.nz

jaganadhg said, at 2010-04-05T07:04:05.000Z:

I am interested in Python3.x port of NLTK

Jaganadh G (Jaggu)

stephenshaw.mail said, at 2010-07-30T12:52:53.000Z:

Count me in.

Stephen Shaw
Trinity College Dublin

adam.morris.329 said, at 2010-08-02T01:46:55.000Z:

I'm definitely interested in helping out.

mcnamara.tim said, at 2010-08-03T04:07:18.000Z:

Numpy is now supports Python 3 http://www.mail-archive.com/numpy-discussion@scipy.org/msg26524.html

StevenBird1 said, at 2010-08-03T06:45:23.000Z:

No numpy distribution yet supports Python 3, as far as I can tell.

alex.rudnick said, at 2011-03-27T21:31:16.000Z:

I'll help too!

StevenBird1 said, at 2011-03-27T21:59:47.000Z:

It would help to know when numpy is likely to support Python 3.x. Here's a stale FAQ post about this: http://new.scipy.org/faq.html#when-will-numpy-support-python-3-x
I wonder if we can get an up-to-date statement about when this is likely to happen.

david.walker64 said, at 2011-03-27T22:17:45.000Z:

NumPy already supports Python 3, according to the NumPy 1.5.0 Release Notes and this document: https://github.com/numpy/numpy/blob/master/doc/Py3K.rst.txt

StevenBird1 said, at 2011-03-27T22:30:01.000Z:

Oh! Then it is high time we ported NLTK to Python 3. Let's plan the work now, but don't begin it until NLTK 2.0 is finalized, within a few weeks I hope. Does anyone have experience with 2to3 conversion?

I propose that we start by assessing the size of the task, by running the 2to3 converter and then running the NLTK tests: cd nltk/test; make
(a complication here is some of the tests break with the current NLTK)
http://docs.python.org/library/2to3.html

Then we can ask for more volunteers if necessary.

bobobo1618 said, at 2011-03-29T23:49:40.000Z:

I started it. I just ran 2to3-3.2, ran setup.py install and corrected the errors that came up. Mostly removing old imports and replacing the nltk elementtree with the standard python one.
I have attached the package (patched version of the release).

bobobo1618 said, at 2011-03-29T23:54:15.000Z:

I also have numpy installed and it works fine :)

StevenBird1 said, at 2011-03-30T01:00:17.000Z:

Nice work, thanks! There's minor changes still to be made to NLTK prior to finalizing 2.0. What do you think about maintaining a nltk3 branch, and tracking the ongoing changes to NLTK? This will facilitate testing by others, as they migrate their own NLTK-dependent code to Python 3.x.

bobobo1618 said, at 2011-03-30T04:24:05.000Z:

I'm not all that experienced with Python... All the stuff I did was pretty basic.
And I'm not too familiar with the details of maintaining a branch (or patches, or working as part of a coding team for that matter). If you can educate me in the workflow I'll do what I can :)

StevenBird1 said, at 2011-03-30T04:42:23.000Z:

Please take a look at the documentation for svn branches, and let me know if you'd like to take this on: http://svnbook.red-bean.com/en/1.1/ch04.html

bobobo1618 said, at 2011-03-30T05:16:28.000Z:

Yeah, seems simple enough to pick up, I'll do what I can.

bobobo1618 said, at 2011-03-30T10:53:08.000Z:

I have the corpus working now. Patched version is attached. Note that the way Python 3 treats bytes and strings means that most of the strings that the corpus functions return are byte strings not normal strings (backslashes are treated as normal characters).
Should I have them all return strings or what?
Btw, sorry about the repackaging of the whole source code. I haven't got SVN checked out yet and haven't used diff on linux yet. This is quick and dirty but it works.
Also keep in mind that this is the nltk-2.0b9/nltk folder, not the nltk-2.0b9 like the old one.

bobobo1618 said, at 2011-03-31T00:55:55.000Z:

Ok, started on the svn branch.
Patch is attached. 2to3 seems to want to remove all the static stuff from wordnet for some reason... As a result I didn't let it get to it.

david.walker64 said, at 2011-03-31T01:02:42.000Z:

Regarding the corpus functions, wouldn't it be best to have them return Unicode strings?

StevenBird1 said, at 2011-03-31T01:17:31.000Z:

You now have commit access, so please branch trunk/nltk into branches/nltk3. Yes to Unicode strings, thanks.

bobobo1618 said, at 2011-03-31T01:35:09.000Z:

Righty. I'm committing now. I've fixed most of the corpus functions (the ones with errors) so that they output unicode. Those that still worked (readme and raw) haven't been touched and still output byte strings.

bobobo1618 said, at 2011-03-31T02:02:22.000Z:

Oh, I also completely replaced the old (2004) elementree with the new (2008, same license, included with python3 xml module) one.

StevenBird1 said, at 2011-03-31T02:24:55.000Z:

Yes, nltk.etree should go away.

bobobo1618 said, at 2011-03-31T02:43:15.000Z:

Should I completely remove it and rely on the version the user has installed with python?

StevenBird1 said, at 2011-03-31T05:10:58.000Z:

Yes -- etree has been in the standard library since version 2.5. It was just a temporary measure to include etree for the benefit of people using 2.4.

bobobo1618 said, at 2011-03-31T05:59:50.000Z:

Ok, replaced elementtree, started running doctests, replaced the P4DTI coverage.py with the new one, ran 2to3 over it.
Now the doctests want epydoc which seems to be 2 years dead.

What version number should I give this? 3.0.?

bobobo1618 said, at 2011-03-31T06:01:22.000Z:

I'mm gonna give this 3.0.1 for now...

StevenBird1 said, at 2011-05-03T12:38:44.000Z:

Let's go ahead and do a beta release soon.

bobobo1618 said, at 2011-05-03T13:02:59.000Z:

Heeeellll no :/

None of the graphical stuff works well enough for beta D:

StevenBird1 said, at 2011-05-03T13:26:33.000Z:

OK. You might like to invite help on nltk-dev.

Also, an alpha release without the graphical stuff would still be useful. The graphical stuff is mostly for teaching purposes.

bobobo1618 said, at 2011-05-03T14:23:14.000Z:

I might do that, I'm not amazing with the GUI stuff but I'm pretty sure Tk won't cut it anymore :/

I'll get to it soon :)
I haven't had all that much time since the holiday :/

xim@8d.no said, at 2011-06-01T05:17:04.000Z:

A working beta of matplotlib for python 3 is maintained at https://github.com/matplotlib/matplotlib-py3, if anyone wants to test that.

I imagine NLTK 2.x and NLTK 3.x will be maintaied quite a while. Is there any plan for ensuring feature updates and bugfixes get applied to both branches? E.g. refactoring all code in NLTK 2.0 so it can be automatically translated by 2to3? With this prequisite and a script for handling remaining updates (e.g. removing the ElementTree implementation) the 3.x branch could follow the 2.x branch by an automated process.

Just throwing out an idea :)

bobobo1618 said, at 2011-06-01T05:41:02.000Z:

Already got numpy, scipy and matplotlib (same one you linked, DAMN that was a bitch to find the first time) working under python 3, NLTK dispersion plots and such work, if anyone's interested, the nltk3 branch should actually be kinda readyish for testing. I've been using it for some basic stuff anyway.

I haven't really had the time (motivation) to religiously merge changes with the new branch like I should have but my process is basically, merge the files, run 2to3 on the changed files, do some basic crap to make sure they still kinda work.

xim@8d.no said, at 2011-06-01T06:21:01.000Z:

Awesome, will test your branch tomorrow :)

I just noted, building numpy for py3k, that they have not made a seperate branch. They run 2to3 build-time, having one version number that works for both versions of python.

python2.x setup.py build -> build for python2.x
python3.x setup.py build -> build for python3.x after running 2to3

bobobo1618 said, at 2011-06-01T06:22:53.000Z:

It's somehow built into distribute, I don't really understand it :/

bobobo1618 said, at 2011-06-01T08:18:48.000Z:

But I'm working on it...

I've merged the trunk with the branch too. Mostly seems to work. The largest source of error I have is the difference between strings and bytes. Mostly all I have to do is add .decode() to .readline() :/

xim@8d.no said, at 2011-06-02T19:37:52.000Z:

The best approach might be changing the original source to using unicode objects everywhere, i.e. decoding all str objects to unicode objects as soon as we can in nltk 2.x trunk. That way we minimize the change required when doing 2to3.

bobobo1618 said, at 2011-06-03T01:44:22.000Z:

I'm thinking of writing a 2to3 fixer and integrating that instead but I don't know. Changing them all to unicode would be better I think.

xim@8d.no said, at 2011-06-03T05:16:21.000Z:

Tested some of the basic stuff, including matplotlib graphs and .download(). The thing I miss the most is some of the corpi. Some don't work because the base CorpusReader uses the 'bytes' data type. All readers should IMO return str objects, anyway. As all of them should have returned unicode objects in 2.x.

The corpus readers are maybe the most critical to a beta or even alpha release. Rewriting the base CorpusReader object to return unicode strings in both versions would be my recommendation, but would be changing the API... An alternative would be to omit the api.py from any 2to3 process and maintain a seperate api.py for python3, with different (simpler, actually!) handling of encoding. This may hold valid for other files as well. Thoughts?

bobobo1618 said, at 2011-06-03T05:55:38.000Z:

The only corpora that have been adapted so far are those that I have personally used for projects, so Brown and the Inaugural Address corpora. It isn't particularly complicated to fix them and those that I just mentioned do return strings as you suggest (Steven told me to) but I haven't got around to the others yet.

I think I fixed some of the stuff in api.py but I can't remember...
It's not really changing the API anyway since we're basically putting it back the way it was and returning strings instead of bytes objects.

xim@8d.no said, at 2011-06-03T19:53:33.000Z:

But I don't think making changes to the string handling in each CorpusReader subclass should be necessary. We just have to make the CorpusReader in our py3 version always return strings, and much of the corpus readers may/should work, and return str objects! IMO this is a better solution than changing a couple of things here and a couple of things there in all the corpus readers. In my opinion the CorpusReader should always have returned unicode objects, but it may be a bit late for that now in NLTK 2.x.

bobobo1618 said, at 2011-06-04T13:03:52.000Z:

It seems I've already changed the CorpusReader to return strings...

xim@8d.no said, at 2011-06-04T19:21:35.000Z:

It's not the CorpusReader itself that returns the bytes, apparently. It's other classes that CorpusReader uses, from what I see the PathPointer subclasses.

We shouldn't use:
open(self._path, 'rb')

but:
open(self._path)

I made a list of all instances of open using 'rb' and 'wb' in the code. Of course, some of them may be best of as 'rb', e.g. those handling IO directly with zip files...

Also, then open() call now takes an argument for encoding. This should make some of the unicode specific stuff obsolete. Like SeekableUnicodeStreamReader(open(source, 'rb'), encoding)) replaceble with open(stream, encoding=encoding)

$ grep 'open(.*rb' `find . -name \*.py`
./test/coverage.py: cache = open(self.cache, 'rb')
./data.py: return open(path, 'rb')
./data.py: self.fp = open(self.filename, 'rb')
./classify/maxent.py: weightfile = open(weightfile_name, 'rb')
./corpus/reader/timit.py: w = wave.open(self.open(utterance+'.wav'), 'rb')
./corpus/reader/xmldocs.py: s = open(fileid, 'rb').readline()
./corpus/reader/util.py: open(self._fileid, 'rb'), self._encoding)
./corpus/reader/util.py: self._stream = open(self._fileid, 'rb')
./downloader.py: file = open(file, 'rb')
./app/wordnet_app.py: f = open("favicon.ico", "rb")
./app/wxwordnet_app.py: pkl = open(pickle_file_name, 'rb')

$ grep 'open(.*wb' `find . -name \*.py`
./test/coverage.py: cache = open(self.cache, 'wb')
./data.py: outfile = open(filename, 'wb')
./classify/maxent.py: trainfile = gzip.open(trainfile_name, 'wb')
./classify/maxent.py: trainfile = gzip.open(trainfile_name, 'wb')
./corpus/reader/util.py: output_file = open(output_file, 'wb')
./corpus/reader/util.py: output_file = os.fdopen(fd, 'wb')
./downloader.py: outfile = open(filepath, 'wb')
./downloader.py: out = open(filepath, 'wb')
./chunk/named_entity.py: out = open(outfilename, 'wb')
./app/wxwordnet_app.py: pkl = open(pickle_file_name, 'wb')

bobobo1618 said, at 2011-06-05T01:41:33.000Z:

Well I'm sure it's like that for a reason, maybe Steven can explain?