import wikiedia:* language tags

git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@28373 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
Add shell-style wildcard match (*,?) do style.

This feature is interesting when hstore is in use.

In this case we can get rid of a huge part of unneded tags like
"tiger:*" created by imports.

See http://taginfo.openstreetmap.org/keys for a list of tags in
order of their occurrence.



git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@28379 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
remove "isvalid" check added in 28329  because it breaks polygons with touching inner rings.
ideally we should drop polygons with problems other than touching inner rings but that code
would still have to be written.


git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@28380 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
[osm2pgsql] Table planet_osm_nodes is hardcoded in middle-pgsql.c

Fixes #4384, reported by kay_D



git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@28386 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
[osm2pgsql] - dynamically allocated memory is not initialized

Use calloc instead of malloc to allocate (with initialisation) the memory of the node-ram-cache

Addresses #4340



git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@28387 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
yet another way to define the iso country code


git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@28388 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
remove outdated gazetteer V1 code,
extend regression tests so that they all can pass


git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@28434 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
make 64bit ID space configurable via ./configure script (option --enable-64bit-ids)

The default is still 32bit IDs.


git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@28436 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
check for geos/version.h, it might be in a different package than geos-config,
also make sure that configure really fails if geos or postgresql libs are missing


git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@28437 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
Implement a flat file persistent node cache                                                                      
                                                                                                  
This implements a flat file persistent node cache for slim mode. It replaces the use of pgsql for node caching.
It stores lat / lon pairs in a large monolithic file ordered by osm id. This allows access to nodes in O(1). Assuming a dense
use of osm_id's (for the full planet this is currently at ~80%), this is the most efficient way to represent the data.

The planet_osm_nodes table with indexes currently takes about 100GB for the full planet. In contrast the flat persistent node
cache file is < 16GB. This should allow the disk caches to be more efficient. Furthermore, each read will load in more useful data.
Both aspects should reduce the number of seeks necessary on disk to retrieve node information.

Therefore hopefully this method should be faster (although I don't yet have benchmarks for this) than storing the data in pgsql.
Particularly diff imports may benefit, as during diff imports typically a low hit rate on the ram cache is achieved.

As the density of valid nodes in the osm id range reduces, e.g. due to only importing extracts, the effectivness of this
method quickly reduces. Therefore it is not well suited for extracts.



git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@28513 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
Use async IO in persistent_cache_nodes_get_list

Issue multiple IO requests for all of the blocks in the node list simultaniously (via async IO).
This should speed up processing as higher queue depths improve throughput.

This uses the GNU C library aio functions. In its current implementation of the library, this is not
"true" async IO at the kernel level, but instead the library implements a thread pool to handle multiple
simultanious requests.



git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@28514 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
Add a little utility program to inspect the flat-node-cache file.

It is a stand alone program that allows to query lat / lon values by
osm node id.

It currently isn't built with the make file and for the moment needs to be compiled by hand.



git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@28515 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
[osm2pgsql] unchecked memory allocation leading to segmentationfault

Some large memory allocations weren't checked for failed allocation. This could lead to segmentation
faults if the system ran out of memory. Instead exit the program gracefully and give some additional
suggestions to the user how to fix the problem.

This closes #4497



git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@28527 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
[osm2pgsql] Fail configure if c++ compiler not found

The configure script was checking for a c++ compiler, but did not fail if none was found.
This then resulted failures later on in the configure script, giving missleading error messages.

This closes #4462



git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@28528 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
[osm2pgsql] Disable async I/O during multi-process operations in the flat node cache

When first performing async I/O in a process and then forking that process, aio_suspend64 blocks indefinately in the child process
in some cases, leading to dead lock.

Fix this issue by disabling async I/O in "going over pending ways/relations" if more than one process is used.

Performance improvements of async over sync I/O appear to be limited anyway, although under ideal benchmark conditions, it did appear to make a 10 - 20% difference on SSDs for a single process. Performance using multiple processes however is much higher than single process async I/O.




git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@28597 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
[osm2pgsql] Add a benchmark testing mode to node-persistent-cache-reader.

node-persistent-cache-reader allows you to query the node-persistent-cache file by node id.

This commit adds a benchmarking mode, in which it randomly requests nodes via the  persistent_cache_nodes_get_list,
testing the read capabilities of the flat-node-cache code and measures the speed of retrieving nodes.



git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@28598 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
use shared memory to join output of parallel processes

git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@28632 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
[osm2pgsql] Bug fix output of "going over pending ways"

also output the number of pending ways / relations before starting to process them to indicate progress



git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@28638 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
[osm2pgsql] Update debian build files to reflect the ppa packages



git-svn-id: http://svn.openstreetmap.org/applications/utils/export/osm2pgsql@28644 b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
Merge remote-tracking branch 'osm/master'