komoot/photon — pull request #112 by lonvia (2014-10-09 19:55:02+00:00)
Title: Improve performance of Nominatim importer

This pull request proposes two changes to the Nominatim importer to make it faster:

* put the writer part into its own thread, so that reading from the Nominatim database
and writing to elasticsearch can be done in parallel.
* query Nominatim's placex/place_addressline table directly instead of going through get_addressdata().
Doing the processing in Java instead of postgres is an order of magnitude faster. It also allows to
cache the frequently used country_name table.

On my planet installation the speed improvement is around factor 2.5. I haven't greatly tested correctness, just checked a few results and they look like they produce the same results as before. It would be great to do another dump (maybe of Germany only to start with) but I can't seem to find the dump tool anymore.

The pull request also contains changes that allow giving the cluster name as a command line parameter. I can move that in a separate pull request if necessary.