kothic/kothic — pull request #6 by M4rtinK (2012-10-16 20:28:22+00:00)
Title: store nodes in sqlite to improve memmory usage

I have modified the import script to store the nodes in temporary a sqlite atabase. It previously just stored them in a dict in memory which caused huge memory usage for large OSM files (such as country-extracts).

The temporary database is indexed by the node id so lookups should be very fast. And because the bulk of the nodes is stored on disk, processing of very big OSM files should now be possible.

From my testing (processing OSM data for Czech Republic) I've got from 9.5 GB down to about 3.5 GB RAM usage.