nltk/nltk — Issue #25: Chunker efficiency
Author: alexrudnick
Created: 2012-01-17 06:13:47+00:00
State: open

Reported by Steven Bird, July 10, 2008

The chunk parser code may have some inefficiencies buried within it...

From Greg Aumann:

grammar_parse takes 21 seconds to parse my lexicon and chunk_parse takes
143 seconds according to these results.

I am attaching four files.
1) parser_profile.py: parses my lexical data using both grammar_parse
and chunk_parse then discards the result. The purpose is just to
generate profiling data.

2) LBDictGram.py contains the two grammars, used by parser_profile.

3) prof_stats.py generates the summary of the profiling stats.

4) profile_results.txt the summary of the profile written by prof_stats.py

The profiling data files written by parser_profile.py are 96 Mbytes each
so I am not mailing those. Also my data is quite large.

Greg

3053795 function calls (3053751 primitive calls) in 20.501 CPU
seconds

Ordered by: internal time, call count
List reduced from 50 to 20 due to restriction <20>

ncalls tottime percall cumtime percall filename:lineno(function)
26 3.447 0.133 20.294 0.781 data.py:96(grammar_parse)
226851 3.379 0.000 10.639 0.000 toolbox.py:79(fields)
226851 2.557 0.000 6.027 0.000 toolbox.py:45(raw_fields)
304154 2.238 0.000 3.468 0.000 sre.py:126(match)
184489 2.065 0.000 3.902 0.000 ElementTree.py:1075(start)
453650 1.231 0.000 1.231 0.000 utf_8.py:15(decode)
304232 1.230 0.000 1.232 0.000 sre.py:213(_compile)
368978 1.167 0.000 1.388 0.000 ElementTree.py:1046(_flush)
184489 0.947 0.000 2.035 0.000 ElementTree.py:1091(end)
184463 0.744 0.000 1.137 0.000 ElementTree.py:285(append)
184489 0.400 0.000 0.400 0.000 ElementTree.py:190(__init__)
184463 0.393 0.000 0.393 0.000 ElementTree.py:726(iselement)
123006 0.257 0.000 0.257 0.000 ElementTree.py:1064(data)
123006 0.222 0.000 0.222 0.000 string.py:308(join)
1 0.168 0.168 20.501 20.501
grammar_prof.py:19(grammar_parse_lbdict_files)
26 0.036 0.001 0.036 0.001 toolbox.py:29(open)
26 0.014 0.001 0.014 0.001 data.py:73(_make_parse_table)
26 0.003 0.000 0.003 0.000 toolbox.py:120(close)
9/3 0.001 0.000 0.001 0.000 sre_parse.py:374(_parse)
26 0.001 0.000 0.001 0.000 posixpath.py:56(join)

19611531 function calls (19554773 primitive calls) in 142.584 CPU
seconds

Ordered by: internal time, call count
List reduced from 441 to 20 due to restriction <20>

ncalls tottime percall cumtime percall filename:lineno(function)
2778182 17.934 0.000 17.934 0.000
sre_parse.py:773(expand_template)
196040 17.774 0.000 49.911 0.000 sre.py:136(sub)
193908 17.120 0.000 33.737 0.000 __init__.py:280(_verify)
7014711 16.677 0.000 16.677 0.000 __init__.py:272(_tag)
2778182 12.551 0.000 30.485 0.000 sre.py:259(filter)
96954 9.433 0.000 14.950 0.000 __init__.py:245(__init__)
290862 6.236 0.000 7.553 0.000 sre.py:154(split)
226851 3.493 0.000 10.802 0.000 toolbox.py:79(fields)
791852 3.402 0.000 3.696 0.000 sre.py:213(_compile)
234335 2.601 0.000 5.020 0.000 ElementTree.py:1075(start)
96954 2.581 0.000 22.632 0.000
__init__.py:316(to_chunkstruct)
226851 2.535 0.000 5.939 0.000 toolbox.py:45(raw_fields)
26 2.385 0.092 21.982 0.845 toolbox.py:135(_record_parse)
96954 2.335 0.000 108.963 0.001 regexp.py:602(parse)
304154 2.153 0.000 3.403 0.000 sre.py:126(match)
468670 2.100 0.000 2.562 0.000 ElementTree.py:1046(_flush)
185401 1.891 0.000 2.557 0.000 ElementTree.py:447(Element)
62395/7458 1.847 0.000 6.902 0.001 data.py:22(_tree2etree)
384521 1.700 0.000 2.500 0.000 ElementTree.py:285(append)
26 1.390 0.053 142.249 5.471 data.py:38(chunk_parse)

Migrated from http://code.google.com/p/nltk/issues/detail?id=25

earlier comments

anand.jeyahar said, at 2011-06-02T05:07:31.000Z:

I am not able to find the profiler files attached. I searched in the nltk-dev google groups too, but couldn't find.. i would like to take a shot at this..