nltk/nltk — Issue #145: nltk.parse.malt should let you specify a pre-trained model when instantiating
Author: alexrudnick
Created: 2012-01-17 06:27:06+00:00
State: open

When instantiating nltk.parse.malt.MaltParser, one might want to use a pre-trained MaltParser model (.mco file), either one that you trained yourself, or one that you downloaded.

And while you can manually set the mco field on the object, that's not exposed in the API, and a use who hasn't dug into the code wouldn't know that.

Additionally, if the user knows the location of the MaltParser jar, they should be able to specify that at construction time too.

Proposed solution: add two parameters ("model" and "malthome") to the __init__ for nltk.parse.malt.MaltParser.

(If the user then goes to re-train the model, then it shouldn't overwrite the existing .mco file.)

I'll put in a patch to do this and email nltk-dev to see if people think this is sensible.

Migrated from http://code.google.com/p/nltk/issues/detail?id=652

earlier comments

hectorvd said, at 2011-07-22T23:28:29.000Z:

Alex, did you get around to commit this patch? I could give it a shot if it's still needed.

alex.rudnick said, at 2011-07-23T00:17:13.000Z:

Hey hectorvd,

I haven't done it yet! If you put in a patch, I'll review it and send it to nltk-dev, though!

Just attach the patch to this issue. (if you don't get around to it, I'll probably do it next week -- thanks for reminding me :) )

hectorvd said, at 2011-07-24T16:58:06.000Z:

There's some weirdness when specifying the model... have a look at this:

$ pwd
/home/hector # <--- current directory

$ java -jar /home/hector/malt-1.4.1/malt.jar -c /home/hector/malt-1.4.1/de-20.mco -i /tmp/malt_input.conll -o /tmp/malt_output.conll -m parse
-----------------------------------------------------------------------------
MaltParser 1.4.1
-----------------------------------------------------------------------------
MALT (Models and Algorithms for Language Technology) Group
Vaxjo University and Uppsala University
Sweden
-----------------------------------------------------------------------------

Started: Sun Jul 24 12:52:35 EDT 2011
The file entry '/home/hector/malt-1.4.1/de-20_singlemalt.info' in the mco file '/home/hector/home/hector/malt-1.4.1/de-20.mco' cannot be loaded.

Even though I'm specifying the model with an absolute path, "-c /home/hector/malt-1.4.1/de-20.mco", maltparser is looking for it in "'/home/hector/home/hector/malt-1.4.1/de-20.mco'".
I guess this could by bypassed using symbolic links, but still seems kinda weird to me.

hectorvd said, at 2011-07-24T17:15:58.000Z:

Ok, here's a first iteration. Let me know what you think. Here's also a test sentence that I used:

* Original: "Die Tür ist nie verschlossen."

* Tokenized: [u'Die', u'T\xfcr', u'ist', u'nie', u'verschlossen', u'.']

* Tagged: [(u'Die', 'ART'), (u'T\xfcr', 'NN'), (u'ist', 'VAFIN'), (u'nie', 'ADV', (u'verschlossen', 'VVPP'), (u'.', '$.')]

* Parsed: (ist (Tür Die) (verschlossen nie) .)

hectorvd said, at 2011-07-24T19:10:20.000Z:

I just noticed that the parsing is really slow (using the test script attached), the times are in seconds. Almost a minute for 10 sentences seems wrong... isn't MaltParser supposed to be really fast (~100 sentences/minute)?

Finished initialization: 0.00908803939819
Finished unicode: 0.00916910171509
Finished tokenization: 0.00929403305054
Finished tagging: 0.133550882339
Finished parsing: 58.4743831158
Finished printing: 58.4753320217

hectorvd said, at 2011-07-24T20:24:06.000Z:

Sorry to hog the thread, but just noticed that MaltParser has an API:
http://maltparser.org/userguide.html#api

it occurred to me that it might be slow because it's basically loading the model for every sentence, so if we find a way to load the model once and then just have it incoming sentences that would improve performance. I've never used Java, though, or communication between Java and Python... some reference material:
http://stackoverflow.com/questions/3652554/calling-java-from-python

alex.rudnick said, at 2011-07-27T06:21:43.000Z:

Hey hectorvd,

Thanks for putting in work on this! I think we should avoid shelling out to "ln -s", because this has to run on a variety of systems, including Windows. (I wonder whether that _execute method works right on Windows, huh... that one's not your problem, though, unless you want to clean it up!)

It might not be so bad to create a temporary directory, cd to it, then copy the Malt file in there, and I think Python has cross-platform ways to do all those things.

Another approach for making it faster could be to have Malt parse all the sentences in one go, so it doesn't have to load the model each time.

I'll take a look at this too, later this week.