dbpedia-spotlight/dbpedia-spotlight — pull request #134 by jodaiber (2013-01-08 22:15:59+00:00)
Title: Indexing and running the database-backed server

Hey all,

I decided to send this as a pull request instead of pushing it to get some input first (@pablomendes and @maxjakob) about my changes to Server.java+SpotlightInterface.java.

I made changes to Server.java and SpotlightInterface.java to allow running both the .properties based "old" system and the database-backed system. SpotlightInterface checks if it is the DB-backed version by calling Server.getTokenizer(). This is kind of ugly, but I think it's the best we can do without re-writing the REST server.

You can now run the server with either

$ java -jar dbpedia-spotlight.jar org.dbpedia.spotlight.web.rest.Server conf/server.properties

or

$ java -jar dbpedia-spotlight.jar org.dbpedia.spotlight.web.rest.Server /data/spotlight/model_nl http://localhost:2222

The DB-backed server has no config other than the "model folder", which contains everything that's needed to run the server (this folder is created by org.dbpedia.spotlight.db.CreateSpotlightModel, this used to be ImportPig.scala):

$ tree /data/spotlight/model_nl/
/data/spotlight/model_nl/
├── model
│ ├── candmap.mem
│ ├── context.mem
│ ├── res.mem
│ ├── sf.mem
│ └── tokens.mem
├── model.properties
├── opennlp
│ ├── chunker.bin
│ ├── pos-maxent.bin
│ ├── sent.bin
│ └── token.bin
├── opennlp_chunker_thresholds.txt
└── stopwords.list

There is a indexing tutorial here:
https://github.com/dbpedia-spotlight/dbpedia-spotlight/wiki/Internationalization-(DB-backed-core)

There is a demo for Dutch here:
http://jodaiber.github.com/demo/

* the Dutch demo is fully in-memory and needs only around 2GB of memory
* the chunker still has some issues with multi-word expressions like Bondsrepubliek Duitsland, we are fixing that currently
* this uses the OpenNLPChunker Spotter, with P(s). The thresholds for annotation are estimated from heldout Wikipedia data
* I will add more Spotters in the next days