DennisOSRM/Project-OSRM — pull request #20 by sivetic (2011-10-14 14:39:15+00:00)
Title: Split PBF Parser into a single Read and a single Parse thread to increase parsing speed.

I noticed extracting was inefficient in the original threading model, and I wasn't making any significant gains by increasing the number of threads to more than 1 (the default). I split the PBF Parser code into two separate threads instead - one thread reads all data from the PBF, and another thread parses all the data. The Read thread places all data into a thread-safe queue. I added a maximum queue size limit to avoid issues with reading all of the data into RAM (data is read quicker than it is parsed). I noticed an improvement of around 30% over original parsing method. Threading is done with Boost threads. I removed the OMP pragmas as they were no longer necessary.

Note - Read/Parse methods are NOT thread safe any longer. Only one thread may Read and Parse at the same time. I experimented with multiple Parse threads, but the improvement was negligible due to greedy locking of the STXXLContainers object. I'm guessing a better locking mechanism may improve the performance, though I did not have time to test it out.