nltk/nltk — Issue #67: import nltk package and subpackages - should be faster / less import->import->import
Author: alexrudnick
Created: 2012-01-17 06:25:17+00:00
State: open

I found nltk great package, professional, stable and with lot of features.

But, when i do:
>>> import nltk

it takes some time. It is nothing critical but when you test applications
again and again, this "slow" import time becomes more and more obvious.
Also, my preassumption is that normal user of nltk will use maybe only 10%
of the whole package in most cases.

Suggestion is to remove from nltk/__init__.py:
import chunk; from chunk import *
import classify; from classify import *
...
import misc
import toolbox

Then import nltk won't be so greedy.

Then documentation for such packages should be changed:

Imported by default - usage:
>>> import nltk
>>> nltk.tokenize

Not imported by default - usage:
When you drop out from nltk/__init__.py line "import tokenize" then use of
"nltk.tokenize" is not valid any more. Then you must use:
>>> import nltk, nltk.tokenize
>>> nltk.tokenize

The only problem I see is, **that existing code will stop to function - but
fix is very easy**.

I also prefer doing this (which I don't expect that should load the whole
nltk package tree):
>>> from nltk.tokenize import PunktWordTokenizer
>>> p = PunktWordTokenizer()

This is more explicit
>>> import this
The Zen of Python, by Tim Peters
...
Explicit is better than implicit.
...

;)

Migrated from http://code.google.com/p/nltk/issues/detail?id=378

earlier comments

StevenBird1 said, at 2009-05-27T07:17:41.000Z:

Loading is too slow, for the reasons you cite. However, the book examples were
greatly simplified when we got "import nltk" to include everything. Its too late to
change that. Here's some other possibilities:

We could be more selective, and cut down the things that are imported by default, so
long as it doesn't break examples in the book. However, this might cause confusion
as users won't be able to guess what's imported by default or not.

We could import lazily [http://peak.telecommunity.com/DevCenter/Importing], though I
don't think this will help for imports like "from nltk.foo import *".

Note that lazy imports would also solve issue 377 and issue 385.

trebor74hr said, at 2009-05-27T19:27:45.000Z:

General rule I would suggest is to depend on smallest set of external packages as
possible. Using new package for this
[http://peak.telecommunity.com/DevCenter/Importing] I wouldn't suggest.

I think that "We could be more selective, and cut down the things that are imported
by default, so long as it doesn't break examples in the book." is good way to do it.

Some implementation "tips" that could be useful:

1. is to use __all__ in package:
from nltk import *
would import everything from __all__ in nltk/__init__.py (see
[http://docs.python.org/tutorial/modules.html#importing-from-a-package]).

2. in order to check source is runnable - you could check examples in book by
running doctest against
[http://docs.python.org/library/doctest.html?highlight=doctest#module-doctest]. If
you could setup book source to be online documentation, base for pdf version of book,
and runnable by doctest for testing, that would be very good. Probably you know about
sphinx [http://sphinx.pocoo.org/] - which could be good way to achive this. Btw. big
python projects like python itself [http://docs.python.org/] and django
[http://docs.djangoproject.com/en/dev/] switched to it.

StevenBird1 said, at 2009-05-27T21:00:57.000Z:

We could easily clone that package for lazy-importing a module; this wouldn't add a
new external dependency. We already use __all__ in our sub-packages, so adding
__all__ to the top-level __init__ file duplicates the manual naming of sub-package
contents while introducing the risk of inconsistencies. We already check book
examples using doctest, not that this would help with the load problem.

trebor74hr said, at 2009-05-28T13:12:01.000Z:

Great to hear that. It seems that lazy imports are not so rare as I thought so:
- there is PEP ralated to this - http://www.python.org/dev/peps/pep-0369/ and I
found some discussion at
http://mail.python.org/pipermail/python-dev/2008-January/076011.html
- from peak.util.imports import lazyModule - which you suggested
- http://pypi.python.org/pypi/zope.deferredimport
- mercurial has it - mercurial.demandimport (source packages
http://www.selenic.com/mercurial/release/?M=D)
- mx.DateTime has it - source preview I found here
http://www.koders.com/python/fidB2399BD41CC244D09D89AD0178CC1D2EAFE9F491.aspx?s=cdef%3Aparser,
download package from here http://www.egenix.com/products/python/mxBase/
- receipe http://code.activestate.com/recipes/473888/

After all, maybe this is not a bad idea ;)

trebor74hr said, at 2009-05-28T13:52:54.000Z:

I haven't checked implementations, I assume that you don't access directly module,
but some proxy object. If this is a case then there are some drawbacks (than I can
now think of):
- dir(module_ref) or module_ref.__dict__ - shows content of proxy object
- sys.modules - doesn't see module until loaded (what is normal)
- del module_ref - should be implemented in proxy to delete ref to loaded module too

Probably there are more issues to consider when choosing the lazy import
implementation, when integrating, and when checking the existing functionality.

StevenBird1 said, at 2009-05-31T11:55:21.000Z:

I've tried a conservative solution, and done some profiling on the "import nltk", and
tweaked some packages. It should take about half as long now. Is it fast enough?

nmadnani said, at 2009-05-31T14:00:53.000Z:

I just checked out the code, rebuilt and tested 'import nltk'. It's certainly faster than it used to be (I also noticed
the sluggishness of that import before).

trebor74hr said, at 2009-06-01T09:25:39.000Z:

I tested svn-trunk r8134, on my (not fast) computer and my impression is that it is
not much faster. But I didn't measured before, so I can't tell you how much faster it
is. So I did profiling now. If you don't something similar - do some profiling before
and after change and compare.

I did it like this (win32):

R:\env-lib\Python25\Lib\site-packages>python -c"import
hotshot;p=hotshot.Profile('r.prof');p.run('import nltk');p.close();import
hotshot.stats;s=hotshot.stats.load('r.prof');s.sort_stats('time');s.print_stats()" >
r.txt

begining of r.txt:
52700 function calls (49147 primitive calls) in 5575.533 CPU seconds

Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
1 300.172 300.172 5515.035 5515.035
r:\env-lib\python25\lib\site-packages\nltk-svn\nltk\__init__.py:15()
1 226.712 226.712 662.565 662.565
r:\env-lib\python25\lib\site-packages\nltk-svn\nltk\corpus\reader\__init__.py:62()
4 207.043 51.761 209.930 52.482
r:\env-lib\python25\lib\sre_compile.py:307(_optimize_unicode)
1 133.302 133.302 246.524 246.524
r:\env-lib\python25\lib\site-packages\nltk-svn\nltk\classify\__init__.py:82()
1 127.026 127.026 529.956 529.956
r:\env-lib\python25\lib\site-packages\nltk-svn\nltk\data.py:32()
1 125.864 125.864 429.106 429.106
r:\env-lib\python25\lib\site-packages\nltk-svn\nltk\tag\__init__.py:13()
1 124.933 124.933 494.726 494.726
r:\env-lib\python25\lib\site-packages\nltk-svn\nltk\internals.py:9()
...

edloper said, at 2009-06-01T12:31:47.000Z:

> 4 207.043 51.761 209.930 52.482 r:\...\sre_compile.py:307(_optimize_unicode)

So 4 calls to re.compile() is taking your system 207 seconds? That seems pretty
extreme to me. When I run the same code I get roughly comparable numbers of function
calls & primitive calls, but the whole import executes in 0.507 CPU seconds (incl
profiling time). That's 10,000x faster than it's executing for you! I wonder what
could cause such extreme differences.

Here's the top bit of profiling information when I run profiling on the import
(horizontally compressed a little to display better):

54228 function calls (50485 primitive calls) in 0.629 CPU seconds

Ordered by: internal time

ncalls tottime percall cumtime percall filename:lineno(function)
4 0.071 0.018 0.072 0.018 py2.5/sre_compile.py:307(_optimize_unicode)
1 0.066 0.066 0.628 0.628 nltk/__init__.py
1 0.053 0.053 0.054 0.054 nltk/sem/relextract.py
833/284 0.029 0.000 0.064 0.000 py2.5/sre_parse.py:385(_parse)
485 0.024 0.000 0.099 0.000 py2.5/sre_compile.py:213(_optimize_charset)
1503/239 0.017 0.000 0.095 0.000 py2.5/sre_compile.py:38(_compile)
1 0.016 0.016 0.016 0.016 py2.5/site-packages/numpy/random/__init__.py
1 0.015 0.015 0.016 0.016 py2.5/lib-tk/Tkinter.py
2084/834 0.014 0.000 0.014 0.000 py2.5/sre_parse.py:146(getwidth)
9662 0.014 0.000 0.014 0.000 py2.5/sre_parse.py:188(__next)
1 0.010 0.010 0.016 0.016 py2.5/pickle.py
1 0.008 0.008 0.055 0.055 nltk/corpus/reader/__init__.py
8282 0.008 0.000 0.019 0.000 py2.5/sre_parse.py:207(get)
235 0.007 0.000 0.008 0.000
py2.5/site-packages/numpy/lib/function_base.py:2851(add_newdoc)
144 0.007 0.000 0.007 0.000 py2.5/textwrap.py:131(_split)
1 0.006 0.006 0.016 0.016 py2.5/site-packages/numpy/core/__init__.py
1 0.006 0.006 0.111 0.111 nltk/parse/__init__.py
1 0.005 0.005 0.021 0.021 nltk/sem/drt.py
1 0.005 0.005 0.043 0.043 nltk/util.py
81 0.005 0.000 0.005 0.000 py2.5/sre_compile.py:264(_mk_bitmap)
1 0.004 0.004 0.060 0.060 nltk/collocations.py
1 0.004 0.004 0.025 0.025 nltk/data.py
635/241 0.004 0.000 0.066 0.000 py2.5/sre_parse.py:307(_parse_sub)
1 0.004 0.004 0.005 0.005 nltk/stem/__init__.py
1 0.004 0.004 0.008 0.008 py2.5/site-packages/numpy/ma/__init__.py

edloper said, at 2009-06-01T12:58:12.000Z:

To get rid of some of the noise, it can be good to profile over more than just one
call to "import nltk". We can do this with:

for i in 1 2 3 4 5 6 7 8 9 10; do echo $i; python -c"import
hotshot;p=hotshot.Profile('r$i.prof');p.run('import nltk');p.close()"; done; cat
r*.prof >all.prof; python -c "import
hotshot.stats;s=hotshot.stats.load('all.prof');s.sort_stats('cum'); s.print_stats()"

(This repeats the import 10 times; more might be better, assuming you're on a
decently fast computer.)

trebor74hr said, at 2009-06-01T14:52:40.000Z:

I'm sorry if I left wrong impression, I didn't wanted to be the person who will test
speed of nltk. I just wanted to point out that profiling tools (e.g. hotshot) can be
useful in finding bottlenecks.

The tests which I runned before were runned on Amd Atlon63 x2 dual 3800+, win xp pro,
but the system is slowed down by antivirus, firewall, etc., so I wouldn't suggest
using these timing data as reference one.

To answer to this:
"That's 10,000x faster than it's executing for you! I wonder what could cause such
extreme differences."
tottime 300 - took 2-2.5 seconds, how you get seconds from these units, I don't
really know (btw. I'm beginner in profiling stuff).

Anyway, I wrote short script for doing loading / unloading for 5 times - with some
hacking for unloading the module (for which I don't guarantee ;) ):

---------------------

import datetime, sys
import hotshot, hotshot.stats
import pdb

def unload_nltk():
# sys.modules says:
# "Note that removing a module from this dictionary is not the same as
calling reload()"
# I read this: it is not forbidden to do this ;)
# see also:
http://stackoverflow.com/questions/437589/how-do-i-unload-reload-a-python-module
for modname in sys.modules.keys():
if modname.startswith("nltk"):
del sys.modules[modname]
for modname in sys.modules.keys():
if modname.startswith("nltk"):
raise Exception("%s still loaded" % modname)

p=hotshot.Profile('r.prof')
for i in range(5):
unload_nltk()
print "%4s end %s" % (i, datetime.datetime.now().isoformat(" "))
p.run('import nltk')
print "%4s start %s" % (i, datetime.datetime.now().isoformat(" "))
p.close()

s=hotshot.stats.load('r.prof')
s.sort_stats('time')
s.print_stats()

--------------

The result is in attachment, It took 9 seconds or 10045 units, and here are first N
lines:

0 end 2009-06-01 16:20:46.206000
0 start 2009-06-01 16:20:48.956000
1 end 2009-06-01 16:20:48.956000
1 start 2009-06-01 16:20:50.472000
2 end 2009-06-01 16:20:50.472000
2 start 2009-06-01 16:20:51.988000
3 end 2009-06-01 16:20:52.003000
3 start 2009-06-01 16:20:53.550000
4 end 2009-06-01 16:20:53.550000
4 start 2009-06-01 16:20:55.050000
186169 function calls (173799 primitive calls) in 17775.934 CPU seconds

Ordered by: internal time

ncalls tottime percall cumtime percall filename:lineno(function)
5 1045.984 209.197 17340.584 3468.117
r:\env-lib\python25\lib\site-packages\nltk-svn\nltk\__init__.py:15()
5 983.621 196.724 3065.661 613.132
r:\env-lib\python25\lib\site-packages\nltk-svn\nltk\corpus\reader\__init__.py:62()
5 772.326 154.465 1267.794 253.559
r:\env-lib\python25\lib\site-packages\nltk-svn\nltk\classify\__init__.py:82()
5 553.259 110.652 4526.779 905.356
r:\env-lib\python25\lib\site-packages\nltk-svn\nltk\parse\__init__.py:42()
5 488.042 97.608 2047.776 409.555
r:\env-lib\python25\lib\site-packages\nltk-svn\nltk\tag\__init__.py:13()
34/5 443.938 13.057 17775.934 3555.187 :1()
5 397.313 79.463 397.313 79.463
r:\env-lib\python25\lib\site-packages\nltk-svn\nltk\metrics\scores.py:10()
5 352.599 70.520 911.559 182.312
r:\env-lib\python25\lib\site-packages\nltk-svn\nltk\metrics\__init__.py:12()
3116/1229 349.706 0.112 861.718 0.701
r:\env-lib\python25\lib\sre_parse.py:385(_parse)
5 332.141 66.428 424.841 84.968
r:\env-lib\python25\lib\site-packages\nltk-svn\nltk\stem\__init__.py:20()

edloper said, at 2009-06-02T03:49:52.000Z:

> To answer to this:
> "That's 10,000x faster than it's executing for you! I wonder what could cause such
> extreme differences."
> tottime 300 - took 2-2.5 seconds, how you get seconds from these units, I don't
> really know (btw. I'm beginner in profiling stuff).

I see -- Windows must be using strange units or something. The first line of your
r.txt file said "5575.533 CPU seconds", which suggests an almost 2-hour import time,
but if it completed in 2-3 seconds then something must be off there. In
linux/unix/os x, the times listed by pstats are all in (CPU) seconds.

I'll continue to do a little more profiling to see if this can be sped up more. One
candidate is precompiling regexps: we precompile ~450 regexps whenever nltk is
imported. That ends up taking up about 25% of the total import time. The modules
that spend the most time compiling regexps (and the amount of time spent, as a
percentage of total time spent compiling regexps), are:

26% python2.5/site-packages/yaml/reader.py
9% nltk/chat/util.py
9% python2.5/tokenize.py
8% python2.5/xml/etree/ElementTree.py
6% nltk/etree/ElementTree.py
6% python2.5/site-packages/yaml/resolver.py
3% nltk/featstruct.py
2% python2.5/HTMLParser.py
2% nltk/corpus/reader/xmldocs.py
1% nltk/grammar.py
23%

curiousdannii said, at 2009-09-21T14:43:25.000Z:

I think the book should really be changed, so that only the minimum is loaded normally.

However, perhaps something like "from nltk.all import *" could be helpful, to load
everything like it does now? "import nltk" could then be optimised for applications
using nltk.

trebor74hr said, at 2010-06-28T08:15:00.000Z:

I like the idea:
"
>> 1. "from nltk.all import *" could be helpful, to load everything like it does now
>> 2. "import nltk" could then be optimised for applications using nltk.
"

StevenBird1 said, at 2011-03-10T01:03:42.000Z:

I'm very reluctant to break lots of examples published in the book in order to solve this problem with slow imports.

I've studied the book examples, and find cases where names from the following packages are presumed to be available at the top level (i.e. nltk.package.name is available as nltk.name): chunk, classify, metrics, parse, tag, tokenize, sem. The remaining three packages that are imported into the top level namespace (inference, model, stem) might be able to be imported on demand.

It would be good if yaml was only imported on demand, to save on the time required to compile all of its regular expressions (reported in comment 13).

I suggest we resolve this issue in NLTK 3.0 by having minimal extra importing there.

StevenBird1 said, at 2011-03-11T04:40:12.000Z:

We can't import yaml on demand, since some of the tagger classes inherit from it (which means they must import yaml) and some taggers are used in the book with just the nltk.name style of import (which means that the top-level init file must import all names from the tag package).

StevenBird1 said, at 2011-10-28T03:27:40.000Z:

I've sought advice for this on the Python list:
http://mail.python.org/pipermail/python-list/2011-October/1282400.html