dakrone/clojure-opennlp — Issue #11: treebank make-tree uses clojure reader, chokes on some tokens from natural language
Author: croeder
Created: 2012-02-26 16:16:57+00:00
State: open

Not a show stopper.

The parsing code here gets treebank strings from OpenNLP. The treebank strings
are very nearly s-expressions and are parsed as such. They are only "very nearly"
s-expressions, not perfectly so because of tokens that are not parsed by clojure.
The code here uses the Clojure reader, so it crashes when it sees a token it doesn't like.
The general idea of going from treebank strings into trees of clojure objects is
still worth pursuing. However, doing it perfectly will require either some pre-processing
or a modified reader.

Not everything that isn't a sequence is a symbol. Not all scalars are symbols. Numbers for example, are happily read by the reader, but are not symbols. That's OK. Some tokens from natural text are not lexed by the reader into clojure. Time values for example, like "2:30". They would appear in the natural language input without quotes. Clojure tries to make things that start with numerals into some kind of number, and the colon throws it off. Since the OpenNLP tokenizer doesn't split 2:30 into 2 : 30, but leaves it, Clojure throws.

My boss at work is a classic AI LISP hack an recommends not using the reader for things that are not lisp s-expressions. He mentioned lisp code he has that basically does the same thing, but can be modified to deal with this case. We work with his academic nephew who is more familiar with the Clojure dialect. He suspects the use of Lisp features not available in clojure. I'll check it out. Hopefully we can get the two of them in on github community fun.
(BTW the features involve macro-related changes to the reader (table?))

It's a plug-in fix. A modified reader would work with the same interface as read-string,
and quote odd stuff like 2:30 that the clojure reader doesn't like...making a string of them.
Using the existing reader for now is fine.