Package: pgsql2osm
pgsql2osmExport .osm from a PostgreSQL openstreetmap databaseMotivationYou have access to a PostgreSQL+PostGIS database which contains openstreetmap
data imported byosm2pgsqlfor rendering maps. How can you generate an extract
to use with other cartography tools ?pgsql2osmis a python script that can generate the XML in.osmformat
for all entities within given boundaries.
It attempts to preserve all data attributes from original osm data:
nodes, ways and relations and all theirosm_ids, all tags, and
interreferences (references to the outside of the extract you choose will be broken,
unless you use--bbox='-180,-89.99,180,89.99'to select all available data).The main usecase I have for this is to generate an.obffile usingosm2obffor the AndroidOsmAndapp offline maps.RequirementsDatabasewas imported withosm2pgsqlin--slimmode (this allows incremental updates and is also leveraged bypgsql2osm.py)Recommended: with--hstorecontaining all remaining tags
not saved as columns in thetags::hstore. If not, the produced.osmwill
have incomplete tags (but will be geometrically sound)with--flat-nodes, and access to the--flat-nodes <FILE>cache binary file
(for now, must be readwrite)The geometry tables are the first with names that end
in_point,_line,_polygonlisted in the following query:SELECT f_table_name,f_geometry_column,srid,pg_size_pretty(pg_table_size(f_table_name::text))
FROM geometry_columns WHERE srid!=0 ORDER BY pg_table_size(f_table_name::text) DESC LIMIT 10;Same for_waysand_relstables:SELECT relname,relkind,pg_size_pretty(pg_table_size(relname::text)),
    (SELECT nspname FROM pg_namespace WHERE oid=relnamespace) AS namespace
FROM pg_class WHERE relkind IN ('r','m','v') AND relnamespace NOT IN (SELECT oid FROM pg_namespace
        WHERE nspname IN ('pg_toast','pg_catalog','information_schema'))
    AND (relname~'_rels' OR relname~'_ways')
ORDER BY pg_table_size(relname::text) DESC;FeaturesXML streaming output, allows to only store compressed .osm.bz2 on disk and not waste RAM.
Example:pgsql2osm --dsn 'dbname=gis' --iso fr --output -|bzip2 > France.osm.bz2Attempt to lower RAM footprint with generators for database queries:
streaming all the way from database to XMLAutomatic detection of table names, referred to here asplanet_osm_*, but they can also
be named anything with the correct suffixes added byosm2pgsql:_point,_line,_polygon,_waysand_relsAutomatic detection of the middle database format (legacy text[] or new jsonb,
available in databases created byosm2pgsql>=1.9). WARNING: very old versions <1.3 may use a
different middle database format, it was undocumented at that time. In that case it is recommended to
re-import with a newosm2pgsqlversion.Automatic detection ofplanet_osm_point,planet_osm_lineandplanet_osm_polygoncolumns: a specific.styleat import is not requiredWarning: The column names you choose will be the keys in the.osmoutput
regardless of the original tag's key. When not using default tag transforms, a.osmfile will be produced that
may not have the tags like they are expected by other software.Note: key-values intagsoverride column key-valuesBounds specifiable aseither:iso country or region code--iso de-byfor Germany/Bayerngrep -i MY regions.csvcan show you recognized iso codes for Malaysia and subregionsosm relation id--osm-rel-id N, for any relation in the database.geojson polygon input file--geojson file.geojsonbounding box--bbox='<lon_from>,<lat_from>,<lon_to>,<lat_to>'Bounds intersection can also be specified as one non-bboxof the above and a--bbox.
The extracted region will then be the intersection (logical AND) of the shape with the bbox.Anti-Feature: unsorted ids, seeUnsorted idsBenchmarksExtractFile sizetimeelements countRAMAustria-left15_0.osm.bz2--bbox='-180,-89,15.01,89'681MBreal 2h18user 37minsys 42min46.7M n4.54M w75K r10.6GBAustria-right15_0.osm.bz2--bbox='14.99,-89,180,89'502MBreal 1h23user 23minsys 17min33.1M n4.01M w67K r-Baden-württemberg.osm.bz2--bbox='-180,-89,180,48.31'274MBreal 49minuser 16minsys 5min17.2M n2.53M w32.6K r-Baden-württemberg.osm.bz2--bbox='-180,48.29,180,89'560MBreal 2h08user 34minsys 21min34.0M n5.84M w63K r6.1GBAustria-left14.25.osm.bz2--bbox='-180,-89,14.26,89'509MBreal 1h21user 21minsys 22min34.6M n3.31M w53K r5.7GBAustria-right14.25.osm.bz2--bbox='14.24,-89,180,89'676MBreal 2h11user 30minsys 46min45.4M n5.27M w89K r10.7GBSwitzerland.osm.bz2692MBreal 2h35user 39minsys 45min47.1M n5.3M w94.5K r-Germany-bavaria.osm.bz21.1GBreal 5h17user 1h06sys 1h3570.1M n10.5M w95.8K r12.1GBNote: The File size is for the compressed extract, bzip2 default settings used unless
otherwise notedRAM usage monitoringI just run the following to get the max RAM used by that processmaxmemkb=0;while sleep 1;do
  memkb="$(ps -Ao rss,cmd|grep python3|grep pgsql2osm|cut -d' ' -f1)";
  if [ "${maxmemkb}" -lt "${memkb}" ];then maxmemkb="${memkb}";fi;
  printf '\033[2K\r%s max=%s' "${memkb}" "${maxmemkb}";doneDisclaimersOutside of bounds geometriesThe bounding region will include every entity and all its dependent entities.
This also means some entities outside the boundary will be inlcuded, as long as
they have any reference to entities within the boundary.
No geometry clipping is taking place, entities are exported as-is and can extend
hundreds of kilometers beyond the boundary if that is a continuous entity
(eg a long river).Unsorted idsThe output ids are streamed in the order they come from the database, and for speed reasons
this is unsorted. If you need a sorted.osm, useosmiumafter exporting:osmium sort in.osm.bz2 --output=out.osm.bz2Database does not store all osm data strictlyThe output data may have some missing tags, because the database import
is not a lossless operation: importing a.osm.pbfto database, then exporting
it to.osmand then converting it back to.osm.pbfwill most probably
(on real world data) produce adifferentfile.Installationpgsql2osmpip install pgsql2osmget_lonlatutilityNeed to compileosm2pgsql(instructions)then copyget_lonlat.cpptoosm2pgsql/src/get_lonlat.cppand in theosm2pgsql/build/directory:makeg++ -std=c++17 ../src/get_lonlat.cpp ../src/node-persistent-cache.cpp \
-I ../contrib/fmt/include/ ../build/src/libosm2pgsql.a -Wall -g -o get_lonlatTest:./get_lonlat /path/to/planet.bin.nodes <<< 3546766428Should output39.188565;49.885284;3546766428if that node was covered by database importRunpgsql2osm /path/to/get_lonlat /path/to/planet.bin.nodes --helpFor an overview of optionsPython moduleMade easier with the ModuleSettings class, example:import pgsql2osm
m=pgsql2osm.settings.ModuleSettings(
  bounds_rel_id=osm_rel_id,
  get_lonlat_binary='/path/to/get_lonlat',
  nodes_file='/path/to/planet.bin.nodes',
  # no need for the --dsn if access already from somewhere else
  access=already_previously_obtained_dbaccess)
with bz2.open('test.osm.bz2','wb') as f:
  m.out_file=f #set the output file object
  m.main()Note: will still print progress reports to stderrImplementation detailsDatabase detailsosm2pgsql's--slimmode means: has the "middle" layout tables containing all the raw osm references betweenways->nodesandrels->nodes,ways,rels, more specifically:The "middle" database tables have names ending in_waysand_rels(osm2pgsqldefaults),
and are the biggest with that suffix in the database, by used spaceThe*_waystable has columnsid::bigintfor the osm ID,nodes::bigint[]for a list of the way's child nodes,andtags::text[]for that way's tags in hstore format,withtags::jsonbin the case of the new jsonb format.The*_relstable has columnsid::bigintfor the osm ID,in the case of legacy text[] format:parts::bigint[]any ids of (mixed) nodes,
ways or relations that are children of that relation,tags::text[]for the relation's tags in hstore format,members::text[]that more precisely describes the children,
example{'n123','admin_centre','w345','outer','w567','inner'}having the node123as a roleadmin_centreand the ways345and567,
as rolesouterandinnerboundaries respectivelyin the case of new jsonb format:members::jsonba list of json objects describing the members, the example continued[{"ref":"123","role":"admin_centre","type":"N"},{"ref":"345","role":"outer","type":"W"}, {"ref":"567","role":"inner","type":"W"}],andtags::jsonbfor the relation's tags in{"key1":"value1","key2":"value2"}format,Access needed:SELECTgranted on the above mentioned tables and also
(for the autodetections of table names and column names)pg_attribute,pg_tables,pg_type,pg_class, andgeometry_columns.Regions.csvTo generateregions.csv, run in a planet database:psql -d gis -p 5432 --csv -c "select -osm_id as osm_id,coalesce(tags->'name:en',name) as name,
    tags->'ISO3166-1' as iso_country,tags->'border_type' as border_type,
    tags->'ISO3166-2' as iso_subcountry,admin_level,tags->'wikipedia' as wikipedia
  from planet_osm_polygon where osm_id<0 and admin_level in ('1','2','3','4')
    and boundary='administrative'" > regions.csvNote: onlyadmin_level<=4 is loaded intoregions.csv.
Use a search onopenstreetmap.orgfor smaller regions.
And then extract the relation id from the url:https://www.openstreetmap.org/relation/5170151701,
eg Switzerland