[osm2pgsql] Be more efficient with virtual memory to reduce chances of out-of-memory

Playing virtual memory tricks works less well than hoped and is fairly OS / kernel dependent.

When using the optimized node cache strategy, no longer allocate twice as much virtual memory
as the -C parameter specifies. Instead use the same memory allocation both for the sparse and
dense node cache. Now, the sparse cache grows from the bottom up and dense cache from the top down.

It still relies on VM tricks (copy on write and overcommit) for forking helper processes though.



git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@27003 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
[osm2pgsql] Bugfix: Node geometries were rounded to 2 s.f. which is incorrect in latlong projection



git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@27004 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
[osm2pgsql] #4061 osm2pgsql-no-return-in-nonvoid-function



git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@27007 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
[osm2pgsql] Don't exit when forking of helper processes fails

Due to the high virtual memory usage of osm2pgsql, too often the forking of
the helper processes to go over pending ways/relations fails. Instead of treating
this as a fatal error, simply fall back to a single process and continue as normal.




git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@27012 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
do not use FIXED_POINT by default as it breaks -l (ell)


git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@27029 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
use FIXED_POINT after all.


git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@27030 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
do not require dpatch; change help text


git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@27036 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
[osm2pgsql] - Change default to not use parallel processing

There still seem to be some issue with the parallel processing under some circumstances,
so change the default to disable it to minimize risk of causing trouble.




git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@27053 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
don't try to find ways in output when there's no index.


git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@27138 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
[osm2pgsql] set synchronous_commit to off

The introduction of the parallelisation during the "going over pending ways" phase
required to not run this phase in an extended transaction, as it would otherwise
deadlock. As now each pending way is processed in a separate transaction, the commit
rate of postgresql became critical for performance. If run with default postgresql parameters
of fsync=on synchronous_commit=on, it resulted in a massive performance degredation.

See report on dev ( http://lists.openstreetmap.org/pipermail/dev/2011-December/023843.html )

Setting the parameter synchronous_commit to off from within postgresql can recover most of this performance.

As all data is derived data from the main osm data, the potential loss of transactions during a crash of the database is
acceptable. Full imports will need to be restarted from scratch anyway, and diff imports should recover on the next diff import.



git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@27179 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
[osm2pgsql] Attempt to improve the transaction management during "going over pending ways/relations"

- Using an extended transaction on the planet_osm_ways table during the pending ways phase should be fine.
  It is the planet_osm_rels table that needs to stay out of transaction in this phase, as the different pending
  ways processes can simultaniously update the pending status of the same relations if multiple ways occur in a relation

- Remove the extended transactions in the going over pending relations phase on the output tables.
  The processing of relations during diff-import causes the ways to be removed from the output tables.
  If a way is contained in two relations in the same diff, that also happen to be processed in different
  parallel processes, this could dead lock.



git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@27182 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
[osm2pgsql] fix and slightly improve debug output for going over pending ways/rels



git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@27183 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
[osm2pgsql] Fix expiry generation with --number-processes > 1



git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@27274 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
[osm2pgsql] - small fix to install-postgis-osm-db.sh script



git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@27277 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
[osm2pgsql] - Create a conditional index on planet_osm_polygon where building is null

Over time, the planet_osm_polygon table has increased in size considerably to currently 17Gb for the full import.
During low zoom rendering, a good portion of time is spent with polygon querries. 
Roughly 80% of these polygons are buildings.

Therefor adding an index on the geometry column for when building is null, can help speed up low zoom rendering by quite a bit.
However, to fully take advantage of this index, the style sheets need to take it into account.



git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@27278 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
gazetter should actually commit its data
(adaption to the changes made for parallelization of postprocessing)


git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@27299 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
[osm2pgsql] Be less aggressive in expiring tiles.

Osm2pgsql's expiry expires a very large number of tiles.
- Reduce the buffer of expiry to 0.1 from 0.5 tiles.
- Be more fine-grained to expire along a line.

This appears to reduce the number of tiles expired to somewhere on the order
of a half.



git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@27376 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
revert r27278 as it breaks usability for non-osm-standard configurations (that might not have a building column at all)


git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@27407 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
Add support for "real" data type:

This allows for stuff like more realistic rendering of rivers based
on the "width" tag or rendering peaks based on their elevation given
in "ele" tag.



git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@27727 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
support unlogged tables for postgres 9.1


git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@27862 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4