valhalla/mjolnir — pull request #25 by kevinkreiser (2015-01-01 00:14:49+00:00)
Title: Thread Tile Writing and Speed Up Pbf Parsing

This pr adds threaded writing of tiles. Currently its hardcoded to spawn 4 threads. In practice this doesnt seem to do a good job of writing in parallel. Often a thread will finish writing all of its tiles before the next thread starts up. Hopefully I can figure out why and fix that.

I noticed that the bulk of the time spent makign tiles though is spent in parsing the pbf, rather running over the in memory data structure constructed from the pbf file. I added a bit of code to tell the ripped off code that we have to skip the bits we dont need and this significantly sped up the PA import. With the default optimization flags this made a huge difference something like 2m 30s down to 1m 50s. With the -O3 flag turned on the threaded branch is only 7% faster or something like that. I think what we really need to do is make parsing of the pbf threaded and write the parsers ourselves (there is a lot of copying around of data in there).

This branch includes some other fixes I found along the way. There was a bug in the nodetable which I fixed via adding a test (this didn't require me to but was way easier to do once I pulled it out into the library). There were more memory leak fixes done as well.