nltk/nltk — Issue #32: English NER data
Author: alexrudnick
Created: 2012-01-17 06:13:56+00:00
State: open

What data set would you like to have added to NLTK? If you have a particular corpus in mind,
please include information about availability, license, and permission for us to redistribute the
data.

There should be some English named entity data in the NLTK corpus collection. We have the
CoNLL 2002 data for Spanish and Dutch. The CoNLL 2003 data may be suitable:

> Language-Independent Named Entity Recognition at CoNLL-2003
> Notes: This dataset is a manual annotatation of a subset of RCV1 (Reuters Corpus Volume 1).
> The annotation per se is available free of charge (subject to a licensing agreement) from the
> CoNLL site. The raw text of RCV1 documents must be requested from NIST (also free of charge
> and also subject to a licensing agreement).
> http://www.cnts.ua.ac.be/conll2003/ner

http://www.cs.technion.ac.il/~gabr/resources/data/ne_datasets.html

Migrated from http://code.google.com/p/nltk/issues/detail?id=109

earlier comments

ewan.klein said, at 2008-11-27T09:56:58.000Z:

The CoNLL 2003 data might be a good bet, but I haven't got round to looking at it
since the procedure for merging the annotation and the raw data presupposes you have
the Reuters data available as a CD.

The MUC34 data doesn't seem to be in a suitable format. It would be really nice if we
could get hold of a portion of MUC7 from LDC, since it is one of the canonical datasets.

StevenBird1 said, at 2009-02-15T05:10:00.000Z:

We have a trained model for named-entity tagging, so providing a corpus is less
important now.