nltk/nltk — Issue #169: Punkt fails on Python 3.2 due to english.pickle error
Author: alexrudnick
Created: 2012-01-17 06:27:39+00:00
State: open

I am using 3.0.1, as I am testing it with Python 3. With Python 3.1, I have no issues using the Punkt tokeniser. With Python 3.2 however, it gives me this delicious error.

[nltk_data] Downloading package 'punkt' to /home/brendan/nltk_data...
[nltk_data] Unzipping tokenizers/punkt.zip.
Traceback (most recent call last):
File "/home/brendan/qa/lib/python3.2/site-packages/apertium_quality-0.2-py3.2.egg/apertium/quality/corpora.py", line 86, in __init__
self.tokenizer = nltk.data.load('tokenizers/punkt/english.pickle')
File "/home/brendan/qa/lib/python3.2/site-packages/nltk/data.py", line 594, in load
resource_val = pickle.load(_open(resource_url))
File "/home/brendan/qa/lib/python3.2/site-packages/nltk/data.py", line 673, in _open
return find(path).open()
File "/home/brendan/qa/lib/python3.2/site-packages/nltk/data.py", line 455, in find
raise LookupError(resource_not_found)
LookupError:
**********************************************************************
Resource 'tokenizers/punkt/english.pickle' not found. Please
use the NLTK Downloader to obtain the resource: >>>
nltk.download().
Searched in:
- '/home/brendan/nltk_data'
- '/usr/share/nltk_data'
- '/usr/local/share/nltk_data'
- '/usr/lib/nltk_data'
- '/usr/local/lib/nltk_data'
**********************************************************************

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/brendan/qa/bin/aq-wikicrp", line 9, in <module>
load_entry_point('apertium-quality==0.2', 'console_scripts', 'aq-wikicrp')()
File "/home/brendan/qa/lib/python3.2/site-packages/apertium_quality-0.2-py3.2.egg/apertium/quality/frontend/corpus_extractor.py", line 23, in main
ui = UI()
File "/home/brendan/qa/lib/python3.2/site-packages/apertium_quality-0.2-py3.2.egg/apertium/quality/frontend/corpus_extractor.py", line 15, in __init__
self.corpus = CorpusExtractor()
File "/home/brendan/qa/lib/python3.2/site-packages/apertium_quality-0.2-py3.2.egg/apertium/quality/corpora.py", line 90, in __init__
self.tokenizer = nltk.data.load('tokenizers/punkt/english.pickle')
File "/home/brendan/qa/lib/python3.2/site-packages/nltk/data.py", line 594, in load
resource_val = pickle.load(_open(resource_url))
ValueError: invalid literal for int() with base 10: '1165\nI14\nsVsutezo\np1166\nI4\nsVbilling\np1167\nI32\nsVshouting\np1168\nI34\nsVholdings\np1169\nI44\nsVtatum\np1170\nI8\nsVoil-pipeline\np1171\nI32\nsVsubside\np1172\nI32\nsVprovincially\np1173\nI32\nsVarmand\np1174\nI2\nsVbrea'

I'm guessing that the pickle was pickled with Python 3.1, and thus needs to be repickled for Python 3.2.

On a sidenote, is there a specific reason that this file _must_ be a pickle and not just an ordinary Python file?

Migrated from http://code.google.com/p/nltk/issues/detail?id=696