nltk/nltk — pull request #564 by muneson (2013-12-14 19:23:15+00:00)
Title: Transformation-based learning

This is basically the same pull request as https://github.com/nltk/nltk/pull/549, which mysteriously got automagically closed. It has slight emendations after some discussion there and in https://github.com/nltk/nltk/pull/555. (Not sure that a new pull request is the best way to do deal with automagic closings -- but how, otherwise?)

The feature description is really unchanged from https://github.com/nltk/nltk/pull/549. Here are comments to the further changes:

* the package now is called `nltk.tag.tbl`, since everyone seemed happy with the tbl-part of the change. @stevenbird proposed moving it to `nltk.classify.tbl` (as the original suggestion) while still making it accessible from `nltk.tag.brill`. To me, this seems like a good idea. At the moment, only the renaming to tbl is carried out, however, so nothing moved to `nltk.classify` before hearing other views.

The current top-level packages seems to be a mixture of named problems (`tokenize`) and general methods to solve them (`classify.*`) -- it seems difficult to do that consistently.

Below I will still call the package `nltk.tag.brill` to avoid confusion.

* the old code in the `nltk2` subpackage is removed and all references to it. External tests may still choke
on it, of course.

* `application` is renamed `task`, to better reflect the semantics; will hold subclasses of Feature and
sets of Templates specific to a given problem (and problem encoding). @Heatherleaf suggested "feature"
which also seems OKish (not clear that it is an obvious home for predefined templates).

* the demo package is gone and moved up into module `nltk.tag.brill.demo`. More demos are unlikely to be needed.

* it is probably reasonable to call the code "based on previous version"
-- a very rough estimation is that 70% of the code is new. I added
the following header to all files as per suggestion.

Natural Language Toolkit: Transformation-based learning

Copyright (C) 2001-2013 NLTK Project
Author: Marcus Uneson
based on previous (nltk2) version by
Christopher Maloof, Edward Loper, Steven Bird
URL:
For license information, see LICENSE.TXT

* trainer and task subpackages are kept for now. I don't have more time right now, but I will return with stronger arguments a bit later. I disagree with the view that deeper module hierarchies by necessity are less transparent or maintainable than shallower ones. In my view, nltk needs more subpackages, not fewer. Anyway, irrespective of how that is solved, it won't affect the API. The user knows nothing about subpackages and may say for instance

from nltk.tag.tbl import Template, Feature, TaggerTrainer, error_list
class MyFeat1(Feature):
...
class MyFeat2(Feature):
...
templates = Template.expand([MyFeat1, MyFeat2])
...
my_tagger = TaggerTrainer(..., templates)

For the record, here is sample of known TBL problem encodings (Features and Templates), all of which would be rather self-explaining if found in `nltk.tag.brill.tasks`, but much less so in `nltk.tag.brill`:

Part-of-speech tagging Brill (1993c , 1995a)
Unknown word guessing Brill (1994); Mikheev (1997)
Text chunking Ramshaw and Marcus (1995)
Prepositional phrase attachment Brill and Resnik (1994)
Parsing/grammar induction Brill (1996)
Morphological disambiguation Oflazer and Tür (1996)
Spelling correction Mangu and Brill (1997)
Word segmentation Palmer (1997)
Message understanding Day et al. (1997)
Dialogue act tagging Samuel et al. (1998)
Prosody prediction Fordyce (1998)
Ellipsis resolution Hardt (1998)
Word sense disambiguation Dini et al. (1998)
Document format processing Curran and Wong (1999)
Grapheme-phoneme conversion Bouma (2000)
Grammar correction Hardt (2001)
Handwritten character segmentation Kavallieratou et al. (2000)
Regression Bringmann et al. (2002)
Hyphenation Bouma (2003)
Named entity recognition Florian et al. (2003)
Compound segmentation Park et al. (2004)
Disfluency detection Kim et al. (2004)
Semantic role labeling Williams et al. (2004)
Word alignment Ayan et al. (2005)
Information extraction Nahm (2005)
Biomedical term normalization Tsuruoka et al. (2008)
Human activity recognition Landwehr et al. (2008)

* trainer.brillorig is also kept -- I have used that both for reference and pedagogical purposes.

* I included a convenience `_run_doctests.py` to facilitate testing on different systems; should be merged
into the larger test suite later

* Yaml is gone and the demo has an example of simple serialization (with pickle). However, there is no serialization to JSON at the moment and I won't have time to do it in a while. JSON seems to be optional, but nice (https://github.com/nltk/nltk/issues/540). Maybe @jskda could chip in?