nltk/nltk — Issue #60: Punkt model has problems with acronyms
Author: alexrudnick
Created: 2012-01-17 06:25:07+00:00
State: open

>>> sent_tokenizer=nltk.data.load('tokenizers/punkt/english.pickle')
>>> sent_tokenizer.tokenize("John saw Mary in the USA today. John saw Mary
in the U.S.A. today. John saw Mary.")
['John saw Mary in the USA today.', 'John saw Mary in the U.S.A. today.
John saw Mary.']

It never wants to put a break after U.S.A. even when its at the end of a
sentence.

>>> sent_tokenizer.tokenize("John saw Mary in the U.S.A. John saw Mary in
the U.S.A. today.")
['John saw Mary in the U.S.A. John saw Mary in the U.S.A. today.']

We need this working for chapter 3

Migrated from http://code.google.com/p/nltk/issues/detail?id=324

earlier comments

jmhelmhout said, at 2010-02-07T21:21:58.000Z:

First of all, thank you very much for creating the NLTK package. It helps a lot by
working with ML and text.

The boundary detection of Kiss and Strunk.

I encountered a difficulty in a text in which the writer applies lots of e.g. and i.e.

Temporarily, I solved this by creating a domain/author specific list of abbrev_types
(add e.g i.e to the set) . Although not the best solution, it can give a quick help
with texts produced in a certain domain or by a person (e.g., needed for author
identification)

Although this dictionary solution is probably not the best, it will help prevent this
issue.

Hence, in making this more explicit by public method call, one could add a domain
specific list to the abbrev_types.

All the best
Martin Helmhout

jan.strunk said, at 2010-05-11T14:41:58.000Z:

I really like the idea of incorporating additional lexical informatin (such as
abbreviation lists) into a trained Punkt model.

joel.nothman said, at 2010-06-30T05:43:30.000Z:

On acronyms:

It may be because the pickled data has changed, but I can't replicate your first example, Steve.

>>> sent_tokenizer.tokenize("John saw Mary in the USA today. John saw Mary
in the U.S.A. today. John saw Mary.")
['John saw Mary in the USA today.', 'John saw Mary in the U.S.A. today.', 'John saw Mary.']

But the issue is in the second example:

>>> sent_tokenizer.tokenize("John saw Mary in the U.S.A. John saw Mary in the U.S.A. today.")
['John saw Mary in the U.S.A. John saw Mary in the U.S.A. today.']

This isn't actually a problem with acronyms:

sent_tokenizer.tokenize("John saw Mary in Calif. John saw Mary in Calif. today.")
['John saw Mary in Calif. John saw Mary in Calif. today.']

And it's also partially a problem with John:

>>> sent_tokenizer.tokenize("John saw Mary in the U.S.A. He saw Mary in the U.S.A. today.")
['John saw Mary in the U.S.A.', 'He saw Mary in the U.S.A. today.']

This is just a false positive given by the algorithm.

Let's look at the learnt parameters:
* 'u.s.a' and 'calif' are both in abbrev_types.
* 'he' is in sent_starters, 'john' is not ('he' was collocated with a preceding '.' in training).
* None of [('calif', 'john'), ('u.s.a', 'john'), ('calif', 'he'), ('u.s.a', 'he')] are in collocations
* ortho_context['he'] == 46, i.e. it appears capitalised at the beginning or middle of a sentence, and in the middle of a sentence lowercase.
* ortho_context['john'] == 14, i.e. it appears capitalised always.

Given this information, 'he' is likely to start a sentence after an abbreviation, while an abbreviation preceding 'John' will be indeterminately a sentence boundary because John will be capitalised in any case.

Note that Kiss and Strunk's rules for using these features mean that setting ortho_context['john'] = 46 is not sufficient to work. Adding 'john' to sent_starters is.

So as long as NLTK's sentence boundary detection uses a largely-unmodified Punkt, and so long as you train it on a corpus that starts sentences with John much less often than the NLTK book does (yay for domain adaptation problems), I don't think this issue will be fixed.

As to the other posts: Incorporating unlearned information is a great idea, as is incorporating information of properties gleaned from corpora with gold-standard sentence boundaries.

mmmasterluke said, at 2010-09-08T09:05:34.000Z:

I'm working on scientific text.

The tokenizer doesn't recognize things like Fig., Eq., Ref. and al. ('et al.') as abbreviations but as sentence boundaries.

I tried to solve the issue by adding items to abbrev_types as suggested in comment 1 but this has no effect (except for 'al'). Sentences are still split between Fig. and the following number.

I also tried adding ('Fig.', '##number##') to the collocations set but nothing changed.

What's the problem here?

jmhelmhout said, at 2010-09-08T11:43:31.000Z:

This helped for me. I do not know if this is applicable for the current release, because I used the following code a long time ago
----------------------
def extract_sentences(text):
punkttt = nltk.tokenize.PunktSentenceTokenizer()

#list of abbreviations remove last period from abbr.
# example list: ['i.e','e.g']
abbreviations = []

for abbrev in abbreviations:
punkttt._params.abbrev_types.add(abbrev)
return punkttt.sentences_from_text(text, True)
---------------
this returns a generator object
use output.next() to iterate

Hope this helps
Martin

mmmasterluke said, at 2010-09-08T13:17:35.000Z:

Thanks for the reply. I did exactly the same in my first approach but to no avail.

I'm either missing something about how Punkt works internally or there is some technical problem.

It would be great to add patterns that cannot contain a sentence boundary (or those that must, for that matter).

joel.nothman said, at 2010-09-11T09:26:36.000Z:

Yes, I agree it would be useful to add patterns that do not contain a sentence boundary, but it's not part of the original algorithm, and so it's not been implemented (yet).

When adding to abbrev_types, make sure you enter the word:
- lowercase;
- without a following full-stop.

mmmasterluke said, at 2010-09-13T09:42:11.000Z:

I forgot to add some abbrevs in lowercase. When I add 'fig' instead of 'Fig' it works. Thanks!