openstreetmap/osm2pgsql — pull request #187 by pnorman (2014-09-29 19:53:03+00:00)
Title: Conversion to C++

# C++ osm2pgsql rewrite #

This PR is substantial, as this diff is thousands of lines of additions and removals.

## Summary

* Rewrite in C++98
* Uses boost
* Adds unit tests
* Performance increases
* In-memory pending tracker
* New multi backend, which allows separation of middle tables and rendering tables, and rendering tables to be updated on multiple databases at once, with different styles or lua transforms

## C++98 conversion

osm2pgsql has been rewritten in object-oriented C++98 with use of boost libraries. Any server that someone is contemplating running as rendering server should support this, as other components require support of this. In particular, any setup that can compile Mapnik should compile this without issues.

Despite the substantial rewrite, parts of osm2pgsql are still using C idioms such as ``void *`` pointers.

C++ brings substantial advantages for writing and not having to manually manage memory and prevent reading out of bounds.

**Note:** Do a ``make distclean`` when switching to/from this branch

Fixes #156

## Boost

The version requirement of Boost has been kept to one that Ubuntu 12.04 supports out of the box, and any machine able to run Mapnik 2 should have a recent enough version of Boost.

## Unit test
Unit tests have been added for the portions rewritten in the conversion. This unit test framework can easily be extended.

Fixes #186

## Validation
As our goal was not to change the osm2pgsql output, a validation of the output was conducted with a planet from 130904. The rendering tables were checked for number of rows, area/length, and checks were done with selected tags. Way/relations were checked for number of rows, and total number of array members.

The only variation seen was in middle table tag counts, which was expected. To facilitate addition of the multi-backend it became necessary to move discarding delete tags from the middle to backend. This means that the middle tables have a complete representation of OSM data, regardless of style files.

*Note: Final validation is running, but earlier versions checked out and smaller regression test passed.*

## In-memory pending tracker

Previously osm2pgsql stored a pending flag on the middle ways and relations tables. This was undesirable for a few reasons
* The data didn't need to be persisted between runs
* After any run, pending should have been false for all rows
* MVCC wasn't necessary, so we weren't getting any advantages from PostgreSQL
* Loading into the tables with pending=true then updating pending=false on a later stage lead to significant table and index bloat, including a partial index which was 3.6GB after import instead of empty
* Having pending in-memory made parts of threading easier

On a full planet import, the pending information for all of the ways is approximately 30MB, so memory usage is not an issue.

The move from in-db to in-memory for the middle was a gain of 7% on single-threaded middle ways, on top of earlier gains in other stages, and overall gains from having smaller less bloated tables.

Fixes #105

Fixes #111

## Multi-backend
The new multi-backend has been developed for situations involving multiple database servers. It allows the middle slim tables to be in a different database than the rendering tables. This allows the rendering tables to be replicated with PostgreSQL replication without also replicating the slim tables, the rendering database servers to be provisioned with less database disk space, and multiple styles to be updated from one osm2pgsql process.

Because of the increased complexity of configuration when multiple servers are involved, the multi-backend takes a JSON file as a configuration, rather than increasing the already excessive number of osm2pgsql command-line options.

This feature is still in development, and will be of interest to only those running multiple rendering databases.

## Performance

Although the conversion to C++ was not done for performance reasons, in practice the use of standard library data structures allows for easier performance gains. Future optimizations should be significantly easier. Overall, there is a performance gain, probably due to in-memory pending.

Performance tests were carried out on the machine used for previous testing (Hetzner EX40-SSD, i7-4770, 32GB RAM, 2x240GB SSD in RAID0) as well as extracts on a lower-end machine (i5-3570 @ 3.4 GHz, 8GB RAM, 500GB 3.5" 7200RPM HDD). These represent one of the fastest possible machines for osm2pgsql imports, and a typical developer desktop with a single mechanical drive.

Unlike some other results, performance numbers **do** include index creation, as that time could be different thanks to in-memory pending changes.

Versions benchmarked were 8fc89648fa877674e889cd26014815a3f029e039 (current master) and 4b6489a755b031d96eeae7b92befa7904d058f4f. Pending relations are not present in a standard import, so do not impact the benchmarks. *Note: Index numbers are still generating, but should not significantly depend on the changes, and will certainly be done before discussion is complete.*

Command lines used were

* ``osm2pgsql/osm2pgsql --create --style osm2pgsql/default.style --flat-nodes flat.nodes --cache 20000 --slim --number-processes 8 planet-130904.osm.pbf`` for full planet tests.

* ``osm2pgsql/osm2pgsql --create --style ~/osm2pgsql/default.style --cache 6000 --slim --number-processes 4 germany-140601.osm.pbf``

Stage
8fc89648fa877674e889cd26014815a3f029e039
4b6489a755b031d96eeae7b92befa7904d058f4f
Change

i7 SSD
i5 HDD
i7 SSD
i5 HDD
i7 SSD
i5 HDD

Processing	Nodes
759
689
1018
699
+34%
+1.5%

Ways
4315
587
4566
580
+5.8%
-1.2%

Relations
9863
12210
9187
11636
-6.9%
-4.7%

Total
14937
13486
14771
12915
-1.1
-4.2%

Pending Ways
4989
2001
3810
1287
-24%
-36%

Ordering and indexes
9500
27976
###
###
+###%
+###%

Grand Total
29467
43519
###
###
+###%
+###%

*Negative percentages are faster*

Note: Germany only needed 2577MB of cache, but I didn't know this number until after I ran the import, but this doesn't effect the conclusions.

## Known issues

The console output is different, and no better than before.