[osm2pgsql] Disable async I/O during multi-process operations in the flat node cache

When first performing async I/O in a process and then forking that process, aio_suspend64 blocks indefinately in the child process
in some cases, leading to dead lock.

Fix this issue by disabling async I/O in "going over pending ways/relations" if more than one process is used.

Performance improvements of async over sync I/O appear to be limited anyway, although under ideal benchmark conditions, it did appear to make a 10 - 20% difference on SSDs for a single process. Performance using multiple processes however is much higher than single process async I/O.




git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@28597 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
[osm2pgsql] Add a benchmark testing mode to node-persistent-cache-reader.

node-persistent-cache-reader allows you to query the node-persistent-cache file by node id.

This commit adds a benchmarking mode, in which it randomly requests nodes via the  persistent_cache_nodes_get_list,
testing the read capabilities of the flat-node-cache code and measures the speed of retrieving nodes.



git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@28598 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
use shared memory to join output of parallel processes

git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@28632 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
[osm2pgsql] Bug fix output of "going over pending ways"

also output the number of pending ways / relations before starting to process them to indicate progress



git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@28638 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
[osm2pgsql] Update debian build files to reflect the ppa packages



git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@28644 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
[osm2pgsql] Allow osm2pgsql to be compiled on mac osx

As reported in #4544 osm2pgsql was not portable to mac osx.

Add additional configure checks for functions that are not available
on mac osx and either disable the functionality or replace it through defines



git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@28646 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
Keep tags that match --hstore-columns. Fixes #4547.

git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@28671 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
[osm2pgsql] Fix previous commit that caused a segmentation fault with --hstore



git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@28683 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
Fix a logic bug in r28671 that made the return value of pgsql_filter_tags() incorrect when --hstore-column was set.

git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@28686 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
Fix a bug that caused --hstore-match-only to be ignored when --hstore-column is set.

git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@28687 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
Extend regression tests to compare results between runs.

git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@28688 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
[osm2pgsql] Remove debug check from binarysearcharray 
 
The binary search array had a sanity check in the add function 
to check that the ordering invariant holds true. This turned addition 
into a O(n) operation rather than an O(log(n)) 
 
Remove this sanity check as a performance optimisation



git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@28700 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
[osm2pgsql] Replace async I/O in flat-node-cache with posix_fadvise 
 
The idea behind using async I/O was to try and have more than a single I/O operation 
in flight, which can greatly increase performance of raid arrays or SSDs but also has some 
advantages on normal disks. 
 
The glib aio functions however did not work correctly, and were unnecessarily complex. 
 
This patch now replaces all of the async code with a simple call to posix_fadvise(WILLNEED) 
for all of the nodes in a get_nodes_list call. 
 
On my 32Gb SSD, the use of posix_fadvise improves performance of node retrieval from 
3.5k nodes/s to 22k nodes/s under ideal benchmark conditions. In real world usage, 
the speedup is likely to be a lot less, as less I/O can be predicted (most get_node_list calls 
only need to load a single 4k block) and thus less I/O can be parallelised.



git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@28701 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
[osm2pgsql] Improve cache management on initial writeout of flat-node-cache 
 
1) fsync the flat-node-cache file in regular intervals to make sure not too 
   much dirty data is in the disk cache which can then stall the whole system 
   during write out. However, instead of fsync, it uses the Linux specific function 
   sync_file_range 
 
2) After writing out the data, drop the information from the disk cache. 
   If one has enough RAM, then one should rather set the -C parameter high enough 
   which is its own ram cache for nodes. It is then a waste of 16GB RAM 
   to also cache the flat-node-cache in disk cache. 
   If one doesn't have enough cache, then chances are that the wrong 
   data is in the disk cache by the time one needs it again, so it is 
   probably also better to drop it from disk cache



git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@28702 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
[osm2pgsql] Enable 64bit IDs by default

Databases generated in 64bit and 32bit ID mode are not compatible with each other,
including applying diffs.

It is therefore necessary to reload the database to switch from 32bit ID mode to 64bit.

Also bump version number to 0.81.1



git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@28712 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
[osm2pgsql] Add an --exclude-invalid-polygon option 
 
Polygons that don't pass the ST_IsValid test are excluded with this switch



git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@28719 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
gazetteer: add aerialway (closes #4499), 
 move place to extratags only when boudary tag is there (fixes #4558),
 move name:prefix to extratags


git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@28746 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
[osm2pgsql] Implement a ways_get_list function to retrieve multiple ways at once. 
 
For the postgresql back-end this issues all way requests of a relation 
in a single sql call and then matches the result back to the list of 
ways in osm2pgsql. 
 
This saves round trip time / context switches to postgresql and potentially 
allows postgresql to be more efficient in retrieving the data. 
 
On tests with small extracts, this nearly doubled performance of the 
relation processing stage and showed increases of up to 25% for the complete 
import. On larger imports this speedup is likely smaller, but hopefully 
still noticeable. 
 
This function currently does not yet combine loading of nodes from 
multiple ways in ways_get_list, which might in future provide additional 
speed up.



git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@28747 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
gazetteer: ignore buildings without name or address


git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@28748 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
use way_get_list function in gazetteer


git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@28761 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4