nltk/nltk — Issue #123: Adding Japanese tagged & parsed corpora
Author: alexrudnick
Created: 2012-01-17 06:26:36+00:00
State: open

I'd like to see the following two Japanese tagged & parsed corpora added to NLTK corpus set.

NLTK Japanese Corpora
http://lilyx.net/pages/nltkjapanesecorpus.html

1. KNB Corpus (Annotated blog corpus)
2. JEITA Public Morphologically Tagged Corpus (in ChaSen format)

I've already acquired the permissions from the corpus creators and they are freely re-distributable as NLTK corpus samples.

In that page I distribute the raw corpus files along with their corpus reader scripts. And please be noted that availability of these corpora is assumed in the Japanese translation of the NLTK book, to which including these corpora data/readers to NLTK is crucial.

Migrated from http://code.google.com/p/nltk/issues/detail?id=587

earlier comments

StevenBird1 said, at 2010-08-24T04:38:37.000Z:

Thanks -- I'll work on adding the data and corpus readers to NLTK.

hagisan said, at 2010-08-29T23:48:01.000Z:

Hi, when are these data and corpus readers expected to added to NLTK? Thanks!

StevenBird1 said, at 2010-08-30T00:46:28.000Z:

Sorry for the delay. Some minor issues:
* can we name the corpus reader modules knb.py or knbc.py, and chasen.py? (none of the existing corpus reader modules include "corpus" or "reader" in their name)
* i'd like to avoid putting corpora larger than 10Mb into the repository; for anything much larger than this it would be good to include a subset (see what we've done for Europarl, for example)
* for the zipped corpus distribution files, would you please provide the standard metadata (please see the XML files here: http://code.google.com/p/nltk/source/browse/trunk/#trunk/nltk_data/packages/corpora)
* is there a reason for having so many copyright notices in the corpus (one per file?)

hagisan said, at 2010-09-01T15:29:31.000Z:

Thanks for your feedback.

1. It's totally OK to rename them to knbc.py and chasen.py
2. I've created the sampled version of JEITA corpus (which include both samples from Aozora bunko and Project Sugita Genpaku) http://lilyx.net/media/jeita.tar.bz2
3. I've prepared them: http://lilyx.net/media/jeita.xml http://lilyx.net/media/knbc.xml Do they look OK?
4. In Aozora bunko and Project Sugita Genpaku, each document has to accompany its own copyright notice. I merged all of them into one file (_copyright.html) and included in the tar.bz2 file.

Thanks!

StevenBird1 said, at 2010-09-16T04:33:48.000Z:

I have added the corpora and corpus readers. Some issues with the corpus readers are documented in nltk/test/japanese.doctest (all these tests currently fail).