nltk/nltk — Issue #170: nltk.classify.svm segfaults on names_demo
Author: alexrudnick
Created: 2012-01-17 06:27:41+00:00
State: open

What steps will reproduce the problem?

leon@ross:~/dev$ py
Python 2.6.5 (r265:79063, Apr 16 2010, 13:09:56)
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from nltk.classify.util import names_demo
>>> from nltk import SvmClassifier
>>> x = names_demo(SvmClassifier.train)
Training classifier...
Segmentation fault
leon@ross:~/dev$

What is the expected output? What do you see instead?

Would expect a standard nltk.classify.util.names_demo() output; instead, execution aborts with a Segmentation Fault.

Test code + stack trace follows.
======

leon@ross:~/dev$ cat segf.py
import nltk
from nltk.classify.util import names_demo
from nltk import SvmClassifier
x = names_demo(SvmClassifier.train)

leon@ross:~/dev$ gdb python
GNU gdb (GDB) 7.1-ubuntu
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i486-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/python...Reading symbols from /usr/lib/debug/usr/bin/python2.6...done.
done.

(gdb) run segf.py
Starting program: /usr/bin/python segf.py
[Thread debugging using libthread_db enabled]
[New Thread 0xb7e74b70 (LWP 26653)]
Training classifier...

Program received signal SIGSEGV, Segmentation fault.
0x00390c1c in ?? () from /lib/tls/i686/cmov/libc.so.6
(gdb) bt
#0 0x00390c1c in ?? () from /lib/tls/i686/cmov/libc.so.6
#1 0x00393ecd in free () from /lib/tls/i686/cmov/libc.so.6
#2 0x01ef42ae in optimize_to_convergence (docs=0xab3a150, label=0xae1b418, totdoc=5000, totwords=193, learn_parm=0xae780f0, kernel_parm=0xbfffed58, kernel_cache=0x0, shrink_state=0xbfffea40, model=
0xab48bc0, inconsistent=0xae20240, unlabeled=0xae25068, a=0xae33ad8, lin=0xae50fb0, c=0xae29e90, timing_profile=0xbfffea5c, maxdiff=0xbfffea78, heldout=-1, retrain=0) at lib/svm_learn.c:1605
#3 0x01ef6fc0 in svm_learn_classification (docs=0xab3a150, class=0xab3ef78, totdoc=5000, totwords=193, learn_parm=0xbfffeb24, kernel_parm=0xbfffed58, kernel_cache=0x0, model=0xab48bc0, alpha=0x0)
at lib/svm_learn.c:247
#4 0x01ee5e1d in svm_learn (self=0x0, args=
([(-1, [(119, <float at remote 0x8b8d5bc>), (95, <float at remote 0x8b8d5bc>), (35, <float at remote 0x8b8d5bc>), (175, <float at remote 0x8b8d5bc>), (63, <float at remote 0x8b8d5bc>), (90, <float at remote 0x8b8d5bc>), (187, <float at remote 0x8b8d5bc>), (14, <float at remote 0x8b8d5bc>), (165, <float at remote 0x8b8d5bc>), (40, <float at remote 0x8b8d5bc>), (69, <float at remote 0x8b8d5bc>), (73, <float at remote 0x8b8d5bc>), (143, <float at remote 0x8b8d5bc>), (186, <float at remote 0x8b8d5bc>), (131, <float at remote 0x8b8d5bc>), (7, <float at remote 0x8b8d5bc>), (156, <float at remote 0x8b8d5bc>), (102, <float at remote 0x8b8d5bc>), (76, <float at remote 0x8b8d5bc>), (199, <float at remote 0x8b8d5bc>), (139, <float at remote 0x8b8d5bc>), (109, <float at remote 0x8b8d5bc>), (107, <float at remote 0x8b8d5bc>), (159, <float at remote 0x8b8d5bc>), (100, <float at remote 0x8b8d5bc>), (157, <float at remote 0x8b8d5bc>), (92, <float at remote 0x8b8d5bc>), (192, <float at remote 0x8b8d5bc>), (103, <float at remo...(truncated), kwds={'type': 'classification'}) at svmlight.c:321
#5 0x080e1c68 in do_call (f=
Frame 0x9516ee4, for file /home/leon/Dropbox/dev/nltk/classify/svm.py, line 241, in train (featuresets=[({'has(e)': False, 'has(d)': False, 'count(b)': 0, 'count(u)': 0, 'has(b)': False, 'count(l)': 0, 'count(q)': 0, 'count(n)': 0, 'has(j)': False, 'endswith': 's', 'count(s)': 1, 'count(h)': 1, 'has(h)': True, 'has(y)': False, 'count(j)': 0, 'has(f)': False, 'has(o)': False, 'count(x)': 0, 'has(m)': False, 'count(w)': 0, 'count(z)': 1, 'has(k)': False, 'has(u)': False, 'count(d)': 0, 'has(s)': True, 'count(f)': 0, 'has(q)': False, 'has(w)': False, 'alwayson': True, 'startswith': 'z', 'count(t)': 0, 'count(c)': 1, 'has(c)': True, 'has(x)': False, 'count(v)': 0, 'count(m)': 0, 'has(a)': True, 'has(v)': False, 'count(p)': 0, 'count(o)': 0, 'has(i)': True, 'count(i)': 1, 'has(r)': True, 'has(g)': False, 'count(k)': 0, 'count(y)': 0, 'has(n)': False, 'has(z)': True, 'has(l)': False, 'count(e)': 0, 'has(t)': False, 'count(g)': 0, 'count(r)': 1, 'count(a)': 3, 'has(p)': False}, 'male'), ({'has(e)': True, 'has(d)': F...(truncated), throwflag=0) at ../Python/ceval.c:3965
#6 call_function (f=
Frame 0x9516ee4, for file /home/leon/Dropbox/dev/nltk/classify/svm.py, line 241, in train (featuresets=[({'has(e)': False, 'has(d)': False, 'count(b)': 0, 'count(u)': 0, 'has(b)': False, 'count(l)': 0, 'count(q)': 0, 'count(n)': 0, 'has(j)': False, 'endswith': 's', 'count(s)': 1, 'count(h)': 1, 'has(h)': True, 'has(y)': False, 'count(j)': 0, 'has(f)': False, 'has(o)': False, 'count(x)': 0, 'has(m)': False, 'count(w)': 0, 'count(z)': 1, 'has(k)': False, 'has(u)': False, 'count(d)': 0, 'has(s)': True, 'count(f)': 0, 'has(q)': False, 'has(w)': False, 'alwayson': True, 'startswith': 'z', 'count(t)': 0, 'count(c)': 1, 'has(c)': True, 'has(x)': False, 'count(v)': 0, 'count(m)': 0, 'has(a)': True, 'has(v)': False, 'count(p)': 0, 'count(o)': 0, 'has(i)': True, 'count(i)': 1, 'has(r)': True, 'has(g)': False, 'count(k)': 0, 'count(y)': 0, 'has(n)': False, 'has(z)': True, 'has(l)': False, 'count(e)': 0, 'has(t)': False, 'count(g)': 0, 'count(r)': 1, 'count(a)': 3, 'has(p)': False}, 'male'), ({'has(e)': True, 'has(d)': F...(truncated), throwflag=0) at ../Python/ceval.c:3773
#7 PyEval_EvalFrameEx (f=
Frame 0x9516ee4, for file /home/leon/Dropbox/dev/nltk/classify/svm.py, line 241, in train (featuresets=[({'has(e)': False, 'has(d)': False, 'count(b)': 0, 'count(u)': 0, 'has(b)': False, 'count(l)': 0, 'count(q)': 0, 'count(n)': 0, 'has(j)': False, 'endswith': 's', 'count(s)': 1, 'count(h)': 1, 'has(h)': True, 'has(y)': False, 'count(j)': 0, 'has(f)': False, 'has(o)': False, 'count(x)': 0, 'has(m)': False, 'count(w)': 0, 'count(z)': 1, 'has(k)': False, 'has(u)': False, 'count(d)': 0, 'has(s)': True, 'count(f)': 0, 'has(q)': False, 'has(w)': False, 'alwayson': True, 'startswith': 'z', 'count(t)': 0, 'count(c)': 1, 'has(c)': True, 'has(x)': False, 'count(v)': 0, 'count(m)': 0, 'has(a)': True, 'has(v)': False, 'count(p)': 0, 'count(o)': 0, 'has(i)': True, 'count(i)': 1, 'has(r)': True, 'has(g)': False, 'count(k)': 0, 'count(y)': 0, 'has(n)': False, 'has(z)': True, 'has(l)': False, 'count(e)': 0, 'has(t)': False, 'count(g)': 0, 'count(r)': 1, 'count(a)': 3, 'has(p)': False}, 'male'), ({'has(e)': True, 'has(d)': F...(truncated), throwflag=0) at ../Python/ceval.c:2412
#8 0x080e1bb0 in fast_function (f=
Frame 0x8e86944, for file /home/leon/Dropbox/dev/nltk/classify/util.py, line 182, in names_demo (trainer=<function at remote 0x8dbd064>, features=<function at remote 0x8da58b4>, names=<WordListCorpusReader(_tag_mapping_function=None, _encoding=None, _root=<FileSystemPathPointer(_path='/home/leon/nltk_data/corpora/names') at remote 0x8f78e3c>, _fileids=['female.txt', 'male.txt']) at remote 0x8f7c36c>, random=<module at remote 0xb7fa1d1c>, name='Zuzana', namelist=[('Zacharias', 'male'), ('Aylmer', 'male'), ('Lissie', 'female'), ('Anica', 'female'), ('Waiter', 'male'), ('Moreen', 'female'), ('Grier', 'female'), ('Matias', 'male'), ('Zita', 'female'), ('Elwood', 'male'), ('Dorena', 'female'), ('Hally', 'female'), ('Barris', 'male'), ('Daisie', 'female'), ('Stella', 'female'), ('Harry', 'male'), ('Ambrosia', 'female'), ('Katalin', 'female'), ('Mandy', 'female'), ('Leiah', 'female'), ('Cathi', 'female'), ('Morgan', 'female'), ('Agna', 'female'), ('Anabella', 'female'), ('Felicdad', 'female'), ('Marji', 'female'), (...(truncated), throwflag=0) at ../Python/ceval.c:3836
#9 call_function (f=
Frame 0x8e86944, for file /home/leon/Dropbox/dev/nltk/classify/util.py, line 182, in names_demo (trainer=<function at remote 0x8dbd064>, features=<function at remote 0x8da58b4>, names=<WordListCorpusReader(_tag_mapping_function=None, _encoding=None, _root=<FileSystemPathPointer(_path='/home/leon/nltk_data/corpora/names') at remote 0x8f78e3c>, _fileids=['female.txt', 'male.txt']) at remote 0x8f7c36c>, random=<module at remote 0xb7fa1d1c>, name='Zuzana', namelist=[('Zacharias', 'male'), ('Aylmer', 'male'), ('Lissie', 'female'), ('Anica', 'female'), ('Waiter', 'male'), ('Moreen', 'female'), ('Grier', 'female'), ('Matias', 'male'), ('Zita', 'female'), ('Elwood', 'male'), ('Dorena', 'female'), ('Hally', 'female'), ('Barris', 'male'), ('Daisie', 'female'), ('Stella', 'female'), ('Harry', 'male'), ('Ambrosia', 'female'), ('Katalin', 'female'), ('Mandy', 'female'), ('Leiah', 'female'), ('Cathi', 'female'), ('Morgan', 'female'), ('Agna', 'female'), ('Anabella', 'female'), ('Felicdad', 'female'), ('Marji', 'female'), (...(truncated), throwflag=0) at ../Python/ceval.c:3771
#10 PyEval_EvalFrameEx (f=
Frame 0x8e86944, for file /home/leon/Dropbox/dev/nltk/classify/util.py, line 182, in names_demo (trainer=<function at remote 0x8dbd064>, features=<function at remote 0x8da58b4>, names=<WordListCorpusReader(_tag_mapping_function=None, _encoding=None, _root=<FileSystemPathPointer(_path='/home/leon/nltk_data/corpora/names') at remote 0x8f78e3c>, _fileids=['female.txt', 'male.txt']) at remote 0x8f7c36c>, random=<module at remote 0xb7fa1d1c>, name='Zuzana', namelist=[('Zacharias', 'male'), ('Aylmer', 'male'), ('Lissie', 'female'), ('Anica', 'female'), ('Waiter', 'male'), ('Moreen', 'female'), ('Grier', 'female'), ('Matias', 'male'), ('Zita', 'female'), ('Elwood', 'male'), ('Dorena', 'female'), ('Hally', 'female'), ('Barris', 'male'), ('Daisie', 'female'), ('Stella', 'female'), ('Harry', 'male'), ('Ambrosia', 'female'), ('Katalin', 'female'), ('Mandy', 'female'), ('Leiah', 'female'), ('Cathi', 'female'), ('Morgan', 'female'), ('Agna', 'female'), ('Anabella', 'female'), ('Felicdad', 'female'), ('Marji', 'female'), (...(tru---Type <return> to continue, or q <return> to quit---
ncated), throwflag=0) at ../Python/ceval.c:2412
#11 0x080e2807 in PyEval_EvalCodeEx (co=0x8da1ec0, globals=
{'names_demo': <function at remote 0x8da5994>, 'LazyMap': <type at remote 0x82c34fc>, 'apply_features': <function at remote 0x8da57d4>, 'log_likelihood': <function at remote 0x8da5844>, 'names_demo_features': <function at remote 0x8da58b4>, '__builtins__': {'bytearray': <type at remote 0x826e640>, 'IndexError': <type at remote 0x822b160>, 'all': <built-in function all>, 'help': <_Helper at remote 0xb7f818ac>, 'vars': <built-in function vars>, 'SyntaxError': <type at remote 0x822ade0>, 'unicode': <type at remote 0x8236f80>, 'UnicodeDecodeError': <type at remote 0x822b5c0>, 'isinstance': <built-in function isinstance>, 'copyright': <_Printer(_Printer__data='Copyright (c) 2001-2010 Python Software Foundation.\nAll Rights Reserved.\n\nCopyright (c) 2000 BeOpen.com.\nAll Rights Reserved.\n\nCopyright (c) 1995-2001 Corporation for National Research Initiatives.\nAll Rights Reserved.\n\nCopyright (c) 1991-1995 Stichting Mathematisch Centrum, Amsterdam.\nAll Rights Reserved.', _Printer__lines=None, _Printer__name='co...(truncated), locals=0x0, args=0x829bc48, argcount=1, kws=0x829bc4c, kwcount=0, defs=0x8da4958, defcount=1, closure=0x0) at ../Python/ceval.c:3000
#12 0x080e0c8b in fast_function (f=Frame 0x829bb0c, for file segf.py, line 4, in <module> (), throwflag=0) at ../Python/ceval.c:3846
#13 call_function (f=Frame 0x829bb0c, for file segf.py, line 4, in <module> (), throwflag=0) at ../Python/ceval.c:3771
#14 PyEval_EvalFrameEx (f=Frame 0x829bb0c, for file segf.py, line 4, in <module> (), throwflag=0) at ../Python/ceval.c:2412
#15 0x080e2807 in PyEval_EvalCodeEx (co=0xb7f79ec0, globals=
{'names_demo': <function at remote 0x8da5994>, '__builtins__': <module at remote 0xb7fa1074>, '__file__': 'segf.py', '__doc__': None, '__name__': '__main__', 'SvmClassifier': <type at remote 0x8deef84>, '__package__': None, 'nltk': <module at remote 0xb7fa15fc>}, locals=
{'names_demo': <function at remote 0x8da5994>, '__builtins__': <module at remote 0xb7fa1074>, '__file__': 'segf.py', '__doc__': None, '__name__': '__main__', 'SvmClassifier': <type at remote 0x8deef84>, '__package__': None, 'nltk': <module at remote 0xb7fa15fc>}, args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../Python/ceval.c:3000
#16 0x080e2907 in PyEval_EvalCode (co=0xb7f79ec0, globals=
{'names_demo': <function at remote 0x8da5994>, '__builtins__': <module at remote 0xb7fa1074>, '__file__': 'segf.py', '__doc__': None, '__name__': '__main__', 'SvmClassifier': <type at remote 0x8deef84>, '__package__': None, 'nltk': <module at remote 0xb7fa15fc>}, locals=
{'names_demo': <function at remote 0x8da5994>, '__builtins__': <module at remote 0xb7fa1074>, '__file__': 'segf.py', '__doc__': None, '__name__': '__main__', 'SvmClassifier': <type at remote 0x8deef84>, '__package__': None, 'nltk': <module at remote 0xb7fa15fc>}) at ../Python/ceval.c:541
#17 0x081005ad in run_mod (fp=0x82e6360, filename=0xbffff5fb "segf.py", start=257, globals=
{'names_demo': <function at remote 0x8da5994>, '__builtins__': <module at remote 0xb7fa1074>, '__file__': 'segf.py', '__doc__': None, '__name__': '__main__', 'SvmClassifier': <type at remote 0x8deef84>, '__package__': None, 'nltk': <module at remote 0xb7fa15fc>}, locals=
{'names_demo': <function at remote 0x8da5994>, '__builtins__': <module at remote 0xb7fa1074>, '__file__': 'segf.py', '__doc__': None, '__name__': '__main__', 'SvmClassifier': <type at remote 0x8deef84>, '__package__': None, 'nltk': <module at remote 0xb7fa15fc>}, closeit=1, flags=0xbffff37c) at ../Python/pythonrun.c:1339
#18 PyRun_FileExFlags (fp=0x82e6360, filename=0xbffff5fb "segf.py", start=257, globals=
{'names_demo': <function at remote 0x8da5994>, '__builtins__': <module at remote 0xb7fa1074>, '__file__': 'segf.py', '__doc__': None, '__name__': '__main__', 'SvmClassifier': <type at remote 0x8deef84>, '__package__': None, 'nltk': <module at remote 0xb7fa15fc>}, locals=
{'names_demo': <function at remote 0x8da5994>, '__builtins__': <module at remote 0xb7fa1074>, '__file__': 'segf.py', '__doc__': None, '__name__': '__main__', 'SvmClassifier': <type at remote 0x8deef84>, '__package__': None, 'nltk': <module at remote 0xb7fa15fc>}, closeit=1, flags=0xbffff37c) at ../Python/pythonrun.c:1325
#19 0x08100812 in PyRun_SimpleFileExFlags (fp=0x82e6360, filename=0xbffff5fb "segf.py", closeit=1, flags=0xbffff37c) at ../Python/pythonrun.c:935
#20 0x0805de5c in Py_Main (argc=2, argv=0xbffff464) at ../Modules/main.c:572
#21 0x0805d03b in main (argc=2, argv=0xbffff464) at ../Modules/python.c:23

Migrated from http://code.google.com/p/nltk/issues/detail?id=697