nltk/nltk — Issue #61: behaviour of extract_rels unintuitive (plus comments on section 7.1 of the book)
Author: alexrudnick
Created: 2012-01-17 06:25:09+00:00
State: open

What version of NLTK are you using? (See nltk.__version__). Please only
submit bug reports for the current version.

0.9.8 (but this applies also to 0.9.7).

What steps will reproduce the problem? (e.g. include Python source code)

This is more or less the textbook example (from the beginning of ch.7):

IN = re.compile(r'\bin\b')
for doc in nltk.corpus.ieer.parsed_docs():
for rel in nltk.sem.extract_rels('ORG', 'LOC', doc, pattern = IN):
print nltk.sem.show_raw_rtuple(rel)

The above code does exactly the same when IN is defined as follows:

IN = re.compile(r'\bin\b.*')

but not the same when it's:

IN = re.compile(r'.*\bin\b')

That is, it looks like a default ".*" is implicitly added to the end of the
filler regexp IN, but not to the beginning.

What is the expected output? What do you see instead?

I find that default ".*" added implicitly to the end of my regexp surprising.

BTW1: in the original regexp in the book, and also in the IE howto, i.e.,
IN = re.compile(r'.*\bin\b(?!\b.+ing\b)'), the third "\b" is spurious (it
matches exactly the same word boundary as the second "\b" in "\bin\b").
(But I think I've already reported that.)

BTW2: a more general remark: I like other parts of chapter 7, concerning
chunking, but section 7.1 is disappointing in various respects:

- there is no mention on how to actually do NER in NLTK,

- relation exctraction, done with the rather idiosyncratic methods
extracs_rels() and show_raw_rtuple(rel), seems convoluted to me,

- (I have the same impression when reading section 2 of
http://nltk.googlecode.com/svn/trunk/doc/howto/relextract.html, with the
natural concept of the tuple (left context, NE1, filler, NE2, right
context) produced in an apparently roundabout way from triples of pairs of
(context, NE))

- the following bit of code, concerning the Dutch "van", is not explained
(the reader doesn't even know that it's about Dutch, unless (s)he spots and
correctly parses the "ned" in "corpus='conll2002-ned'").

Migrated from http://code.google.com/p/nltk/issues/detail?id=329

earlier comments

Adam.Przepiorkowski said, at 2009-03-04T18:27:07.000Z:

P.S. Maybe it would also make sense to mention in section 7.1. of the book that there
is an implicit "window = 10" argument to nltk.sem.extract_rels(), as otherwise it is
surprising that this command with this regular expression produces such reasonable
results.

StevenBird1 said, at 2009-03-04T19:26:13.000Z:

> I find that default ".*" added implicitly to the end of my regexp surprising.

BTW Python's re.match() tries to match a regexp at the start of its string, as
opposed to re.search(). See Matching vs Searching in
http://docs.python.org/library/re.html; (this doesn't mean extract_rels should
behave like this.)

Adam.Przepiorkowski said, at 2009-03-04T20:59:37.000Z:

I see; I am new to Python, didn't know that. (In this case, maybe extract_rels
should be renamed to match_rels, or so?)

StevenBird1 said, at 2009-05-03T10:20:10.000Z:

Ewan -- would you like to take action on this proposed name change, or shall we leave it?